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Insecure 
about Being 
Unsecure 


NEXT 
Letters oO 


here have been epic battles over whether 

“insecure” or “unsecure” should be used when 

referring to computer security. Granted, those 
epic battles usually take place in really nerdy forums, 
but still, one sounds funny and the other seems to 
personify computers. Whichever grammatical construct 
you choose, the need for security is greater now than 
ever. As Linux users, we need to make sure we're not 
overconfident in the inherent security of our systems. 
Remember, they all have a weak link: us. 

Reuven M. Lerner starts off this issue by showing 
how to test the machine learning model he created 
last month. If computers are only as good as their 
programming, it’s important to test how well the 
code learns. Sometimes coming up with a good test 
is harder than writing the original code! 

Dave Taylor finishes his werewolf-warning 
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system—er, | mean his phases of the moon script this month. When 
you Start writing scripts with Dave, you start to realize all the cool 
additions you can make. Programming Is like that. Dave shows how 
to figure out whether the moon is waxing or waning, which is really 
useful information when you're planning a family camping trip in 
werewolf territory. 

Kyle Rankin continues his series on MCollective, which is server 
orchestration for configuration management systems. If Puppet and 
Chef can’t quite handle the day-to-day needs of your environment, 
using MCollective might fill that need. 

| explore Synology this month, which continues to fill my own needs 
for my home office. I’ve had multiple NAS machines through the years, 
and nothing has been as useful, reliable and performant as Synology. 
If you're looking for a storage device that also handles a plethora of 
server functions, you'll want to check out my column this issue. 

der.hans has an incredible in-depth article this month on using 
password managers. We rely so much on our online accounts, it’s vital 
that our passwords be strong and unique on each site. A password 
manager is quickly becoming the only feasible way to accomplish 
that. der.hans discusses how to manage passwords and retrieve them 
when needed. If password managers aren't convenient, it’s unlikely 
anyone will use them, so learning the nuances of such an important 
technology is worth the effort. 

Jan Newmarch finishes his series on low power wireless this month. 
The use of a low-powered wireless network is a constant reminder of 
how much our world is becoming connected. We’re not far from a time 
when a hacker could infiltrate our toaster to ruin breakfast. Thankfully, 
Jan’s series helps us to create our low powered wireless networks 
intelligently. Everyone should read his series, even if you never plan to 
implement something like 6LOWPAN. 

We love bringing you the latest information from the Linux world, 
whether it’s new products, fun apps or even in-depth security articles. 
Linux always has been at the forefront of computer security, and it’s 
only going to stay there if we take security seriously and don’t assume 
we're safe just because we use open source. If you’re hoping to 
become a better, smarter Linux user, this issue is for you. m 
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Mars Lander Program 

Regarding Dave Taylor’s series on the Mars Lander game in the September, 
October and November 2016 issues: | appreciate the way he builds the 
program over three months in the magazine. Is there somewhere | can 
download the finished program, so | can peruse and play with it? 


—John 


The finished program is available at http://www.linuxjournal.com/files/ 
linuxjournal.com/mars2.txt.—Ed. 


Simple Server Hardening 

| found Kyle Rankin’s “Simple Server Hardening” article in the October 2016 
issue to be very useful, and | also look forward to more such tips, such as 
hardening of services that need to be on—for example, a mail server. 


| have one comment: if we disable passwords altogether and allow access 
using SSH keys only, is there still a need to disable root login over SSH? 
The problem with reaching root via a non-root login (using sudo or SSH) 
is then your root is only as secure as such a non-root login. | personally 
don’t even have a password for root, although | permit keys-only SSH 
logins to root. | would be glad if you could highlight risks with this 
arrangement, as against sudo and so on. 

—Mayuresh 


Kyle Rankin replies: This is a great question. Yes, even if you disable 
password login altogether, you still should disable root login both over 
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SSH and locally, and use sudo instead. The core idea 
with reguiring sudo for root access instead of using 
su is that it makes it easy to revoke any individual 
admin‘s access to root without having to change 
some central, shared password. Even if you require 
SSH keys for login, requiring sudo still provides extra 
protection in the case of a compromised key. 


Attackers who have compromised an admin’‘s 

SSH keys still have one extra barrier to the root 
account: they may be able to ssh in to a server 

as the admin using the admin's key (if the admin 
didn’t take the extra step of password-protecting 
the keys), but the attackers still have to figure out 
the admin's personal password to be able to sudo 
to root. In your scenario, if your personal key were 
compromised, attackers would have direct access to 
root on all of your machines. Scenarios like this are 
why it’s so important never to share personal keys 
between admin, password-protect your keys, pick 
good passwords for sudo, and never allow users to 
sudo up to root without their password. 


Hodge Podge Comment 
Shawn Powers’ “Hodge Podge” article in the October 
2016 issue was an interesting article, as usual. 


In the article, he mentioned several economical 
cloud options. | wonder whether he has evaluated 
SDF.org ( g). There are various tiers 
available to suit various needs, the most basic 
costing only a one-time fee for life on a shared 
cluster. There are VPS options as well at the higher 
end. How do these options figure in comparison 
with others that he mentioned? 

—Mayuresh 
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Shawn Powers responds: I'd never heard of SDF.org, but when | visited 
the site, | couldn't find VPS options. | did see free shell access, which is 
cool, but that’s all | could find. 


Hodge Podge Suggestion 

In Shawn Powers’ “Hodge Podge” article in the October 2016 issue, he 
mentioned his love for Synology and GPS trackers. | think | share the same 
passion. In his quest for fun gadgets and programs, | would like to introduce 
Traccar. | use the free program called Traccar (http://www.traccar.org) to 
monitor the movements of my car. | installed the Traccar server on my 
Synology and the Traccar client on an old unused Android smartphone 
(with GPS). | installed the smartphone in my car and connected the 
smartphone to my car battery. On the smartphone, | installed a firewall 
app to minimize data traffic to and from Google (I'm not interested in 
updates or other stuff). 


When the car is parked in the vicinity of my house, the smartphone uses 
the Wi-Fi of my AP. In this way, | minimize data traffic on my pre-paid 
data bundle. Only when I'm driving around GPS coordinates are sent to 
the Traccar server using the pre-paid data bundle. The client is set to 
an interval of one minute. Depending on the cost of your pre-paid data 
bundle and the amount of travel, you will use up $10 in several months. 


The Traccar server stores all coordinates (of multiple devices if needed) in 
a MySQL database. Via a web interface, you can locate your GPS device or 
see where the GPS device, in my case my car, travels or has traveled to. 


Next, | wrote a C-program that does a MySQL query on the Traccar 
database. Whenever my car starts moving and moves beyond a pre-defined 
offset, the C-program sends an email, notifying me that the car is moving. 
The C-program also emails me when the car is stationary with its present 
position (street name and city). The C-program is running on a Raspberry Pi 
next to another beautiful program: Domoticz (https://domoticz.com). 


There you go—a second life for your smartphone and lots of fun. 


—Roland Horsten 
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Shawn Powers responds: Very cool! | wonder if the program would 
record and upload only while within range of home Wi-Fi. It would be 
a great use for old phones and wouldn’t require any cell service at all. 
Either way, I’Il check it out. Thanks! 


Feedback on “Low Power Wireless: 6LoWPAN, IEEE802.15.4 
and the Raspberry Pi” 

I'm one of the linux-wpan maintainers in the Linux kernel as well as the 
wpan-tools maintainer. 


That this project was picked up for an article series in Linux Journal was 
a really nice surprise [see Jan Newmarch’s articles in the November and 


December 2016 issues, as well as the final article in this issue]. 


Reading the first article, | have a few items | wanted to bring up: 


In the article Jan writes that the at86rf230 module needs to be loaded 
manually. That should not be needed and has not been in my testing so 
far. The device tree overlay he enabled should make sure that the driver is 
auto-loaded as well. Maybe that’s something worth checking in his system. 


In the Python example, he uses TCP. | really would recommend using 
UDP for examples here. In his test setup, it will not make much of a 
difference, but in these networks, it can be really lossy and the TCP 
handshake makes things more complicated. In the end, you normally 
do not need a stream protocol for these types of networks. 


Using UDP for this example has the additional benefit that the kernel 
can apply next header compression (NHC, RFC6282) for UDP. If you 
use the right port range, that can save 2 bytes from the UDP header 
in each packet. (The port range is reduced to only 16 ports, starting 
from port 61617.) 


A good article overall. Thanks for doing it and looking forward to the 
next episodes. 
—Stefan Schmidt 
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Jan Newmarch responds: Thanks Stefan! Yes, you are right on both 
counts. | manually loaded the drivers in order to test if it was working. 
Once confirmed, reboots and the device tree load the drivers as needed. 
| should have mentioned that | loaded them manually only as a test and 
after that it no longer would be needed. 


| was undecided as to whether to use TCP or UDP. UDP Is the protocol 
of choice for low-power systems, but | thought that readers might be 
more familiar with TCP programs. The RPi can handle either; it has 
enough capabilities. | my third article (in this issue), I’m using the CoAP 
library aiocoap, and that is UDP-based. 


Holy Triage, Batman! 

| loved Susan Sons’ “Holy Triage, Batman!” article in the November 2016 
issue. So much good advice! Thank you! 

—Mike 


Erratum 

| just finished reading Reuven M. Lerner’s At The Forge (“Preparing Data 
for Machine Learning”) in the November 2016 issue, which was great and 
had me craving a burrito! But | noticed a typo: in the Resources section, 
the Data Science Weekly newsletter URL should be a .org and not a .com. 
The correct URL is http://datascienceweekly.org. 


—Terrill 


SEND US YOUR PHOTOS 
Send your Linux-related photos to 
\jeditor@linuxjournal.com, and we'll run the best ones here. 


WRITE LJ A LETTER 
We love hearing from our readers. Please send us your comments 
and feedback via http://www.linuxjournal.com/contact. 


RETURN TO CONTENTS 
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Letters Editors’ Choice 


diff -u 
What’s New in 
Kernel Development 


Anshuman Khandual felt that there were a lot of devices with their own 
RAM in the world, and Linux should be able to allocate that memory to 
users as part of the general pool, rather than letting those devices hog it 
all for themselves. 

It turns out to be hard to do this. Devices have to be initialized before 
they can be used, so the general pool of RAM would have to be grown 
by the kernel in a coherent way as new devices came online. Also, unlike 
regular system RAM, devices can removed from the system without 
warning, which poses its own set of problems. 

There always are solutions to these sorts of problems, but they may 
involve unpleasant compromises, such as increased code complexity, speed 
reduction or using some areas of RAM in certain ways but not others. 

Sometimes there also are unexpected opportunities for collaboration 
that emerge from these sorts of proposals. In this case, Anshuman 
found that Jerome Glisse’s work on HMM (heterogeneous memory 
management) might be relevant, because it had the similar issue of RAM 
not always being available to the general pool. Jerome suggested they 
work together on finding a proper solution. 

Dave Hansen also pointed out that the existing projects autonuma 
(automatic non-uniform memory architecture) and hugetlbfs were 
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relevant to Anshuman’s work, because they dealt 
with gathering up available RAM and adding It to 
the general memory pool as well. 

But, not all projects can be merged in this 
way. Sometimes developers will make certain 
assumptions, or cut certain corners, because they 
don’t see any other way to implement the thing 
they need. And, sometimes a developer on a 
similar project will need exactly the thing that the 
first developer had been trying to avoid. That's 
what happened here. David Nellans, who'd been 
working with Anshuman on his code, explained 
that, among other things, they wanted to avoid 
hugetlbfs support and concentrate on getting 
what they needed from THP (transparent huge 
tables) instead. But, Dave objected that this would 
force additional complexity into hugetlbfs—to 
special-case the RAM that Dave and Anshuman 
were working on. 

There was quite a technical discussion, with no 
explicit resolution. This is the type of project that 
typically undergoes a wide-ranging discussion 
that gathers participants from all parts of the 
kernel. Eventually, various big-time kernel hackers 
may step in to adjust folks’ direction and point 
out uncompromisable issues or extremely simple 
alternatives to complex debates. 

A ret2dir attack is when a hostile user writes 
malicious code into kernel memory, hoping to 
trick the kernel into executing that code ata 
future date, like when a function call returns to 
that location. It’s similar to the ret2usr attack, 
in which the same general thing occurs in user 
memory instead of kernel memory. 

Juerg Haefliger recently wanted to implement 
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THEY 
SAID IT 


Wait until it is 
night before 
saying that it has 
been a fine day. 
—French Proverb 


There’s no 

such thing as 
quitting. Just 
sometimes 
there’s a longer 
pause between 
relapses. 

—Alan Moore 


Self-confidence 
is the first 
requisite to great 
undertakings. 
—Samuel Johnson 


A moment's 

insight is 

sometimes 

worth a lifetime’s 

experience. 

— Oliver Wendall 
Holmes Jr. 


Our lives begin 

to end the day 

we become silent 

about things 

that matter. 

— Martin Luther 
King Jr. 
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eXclusive Page Frame Ownership (XPFO), which would prevent a user 
from modifying RAM that hadn't been allocated explicitly to that user. He 
posted some code to do this and had a bit of a technical discussion about 
it with Laura Abbott. There were no serious objections—just suggestions 
regarding portability and locking efficiency. It looks like Juerg’s code will 
go into the kernel at some point. 

Pavel Machek felt that certain Linux systems—particularly 
smartphones—didn’t do a great job when battery power became low. 
The Nokia N900, he said, would run down the battery until it didn’t 
have enough power left to do a clean shutdown. It would just crash. 
Pavel proposed adding code to detect when the battery was too low 
and doing a proper shutdown in that case. 

He posted some code to do this, but initially walked it back when 
Tony Lindgren pointed out that a more configurable patch would really 
be better. Specifically, he said the exact voltage that would trigger a 
shutdown should depend on what would be best for any given system. 
To do this, Pavel replied that a good approach would be to create a 
battery object in the device tree and create configurable properties for 
those objects. 

On second thought, however, Pavel said that maybe configurability 
wasn’t really the thing in this case. The hardware had its own mechanism 
for alerting the operating system that the battery was low or dead, and 
Pavel felt that Linux simply should respect those messages. There was no 
need to second guess the hardware in this case. 

The discussion ended inconclusively, but presumably, some kind of clean 
shutdown will replace the current crash behavior.—Zack Brown 
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Android Candy: 
the Verbification 
of Video Chat 


People who study the 
history of languages (5 [) 
probably will look back OOQG e UO 


at our current time 

and scratch their heads. We keep inventing verbs! First, Google became 
the verb we use for searching. Then, “Facebooking” someone became a 
viable way to contact them. Heck, | forgot about “texting” someone. It 
seems we just keep taking perfectly good nouns and making them verbs. 
We keep verbing all our nouns! But | digress. 

Unfortunately, I’ve noticed a trend where people who want to do 
video chat are starting to say, “Facetime me.” The problem isn’t with the 
terminology, it’s with the proprietary technology. If you’re an Android 
user, you can’t “Facetime” someone. Google Hangouts sort of solves 
that problem, but it’s not as user-friendly as Facetime on iOS is. 

Google hopes to change that with its new Duo app. It’s designed very 
much like Facetime, but it has native iOS and Android apps. In my trials, 
the video quality is fine, so it’s really only adoption that needs to happen 
in order to become the standard across platforms. It’s worth a try, but 
you'll have to convince your Apple friends to install a new app for video 
chatting. Also, “duo” doesn’t really lend itself to verbification, so I’m a 
little worried Google has a non-starter on that note alone! 

—Shawn Powers 
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Listen To Me Cheaply 


| listen to a lot of books. A lot. And honestly, although I’ve written 
about the “Listen” app for audiobooks, | tend to use Audible more 
than anything else anymore. Part of the reason is the Android app 
finally has more fine-grained speed settings. (I prefer around 1.4x 
speed.) iPhone people don’t have that seemingly simple feature. 
Just saying. 

The main reason I’ve been using Audible, however, is the pricing. 
Yes, I'm a Platinum subscriber, so | get two credits a month. But, it’s 
the “add narration” feature on the Kindle store that really does it 
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though. It’s often cheaper to buy the Kindle version of a book and 
add Audible narration for less than the cost of the audiobook alone! 
That's not always the case, but it’s always worth checking. Look at 
my example in the screenshots. 

Yes, this book is free. But, you can add Audible Narration for $2.99. 
Notice how much it would cost to buy the book outright: $21.99! 
Not all books come with such a good deal, but many do. And if you 
add Audible narration, the book is added to your Audible account like 
any other book. You never have to read the Kindle version, but it’s 
still nice to have both—especially when it’s cheaper than just buying 
the audiobook! —Shawn Powers 
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Never Trust 
Yellow Fruit 


You've probably heard about the 11 Fu 
WiFi Pineapple from Hak5. It’s a WiFI Pines ] lp 
fascinating device that allows you : HI) 

to do some creepy pen testing. It's 

the sort of tool that could be used for evil, but it’s also incredibly useful 
for securing networks. 

The hardware is fairly basic and resembles an off-the-shelf router. The 
multiple network interfaces really shine, however, when paired with the 
operating system. The WiFi Pineapple software creates a rogue, hidden 
access point that purposefully tricks clients into connecting to /t instead 
of the AP they’re usually connected to. 
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It doesn’t stop there, either. The software sniffs the wireless network for 
clients looking for APs they know, and then creates fake SSIDs that match 
what clients are looking for. Once the client associates with the SSID it 
thinks it already knows, the Pineapple provides internet service, but it also 
sniffs every packet along the way. Scary stuff. 

There are many uses for the WiFi Pineapple, and many of them are 
white-hat sorts of things. Be warned though, because you might be 
connected to someone else’s WiFi Pineapple right now! To get your own 
piece of dangerous awesomeness, head over to http://wifipineapple.com 
now. There is a 2.4GHz-only verison as well as a dual-frequency model. 
—Shawn Powers 
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Analyzing Videos 
for Fun and Profit 


People’s phones and all of the various sensors that may be built in 
to them is a source of scientific data logging that almost everyone 
carries around. Although the selection of sensors varies from phone 
to phone, they almost all have a camera. In this article, | take a look 
at a piece of software called Tracker that can be used to analyze 
videos you take of experiments. 

You can download Tracker directly from the project page: 
http://physlets.org/tracker. A lot of good documentation is available 
at the main website, including examples of how you might be able 
to use it from the community of other users. Tracker is written in 
Java, however, so you also need to have a JVM installed on your 
system before you can use it. For example, on a Debian-based system, 
you can install a very good JVM with the command: 


sudo apt-get install openjdk-8-jre 


You then can download the relevant installer for your system and run 
it from a terminal window. You probably will need to make the installer 
executable with a command like this: 


chmod +x Tracker -4.95-linux-64bit-installer.run 


Don't forget to use the sudo command so that you have the correct 
permissions to do the installation. 

Once it’s installed, you should have a new entry in your application's 
menu system. Starting Tracker will give you a new empty project for 
beginning your video analysis. 

Although you could start analyzing your own videos immediately, you 
may want to use simpler videos while learning how Tracker works. When 
you ran the initial installer for Tracker, it should have asked you whether 
you also wanted to install example files. Assuming you said yes, you now 
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Figure 1. 


File Edit Video Track Coordinate System View Help 
| 9] P-L | create = | Qt) | \ % AALS Al 
Vv 


x«& 43 ¢| Whenyou start 
memory in use: 20MD of 247MD 


ke a 
Plot view of track data will appear here. anew em pty 


Tracker, you get 
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project. 


Table view of track data will appear here. 


Figure 2. 

The open file 
dialog box 
allows you to 
open movie files 
or Tracker files. 


File Name: | 
Files of Type: |Video and Tracker Files 7 


cnc 


can load one of those within Tracker. When you click on the menu item 
File—Open File, you'll see a new window where you can select either a 
movie file or a Tracker file. 

To begin with, let’s look at one of the Tracker files from the experiments 
folder. In Figure 3, | have loaded the experiment file named BallToss.trk. 
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File Edit Video Track Coordinate System View Help 
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Figure 3. Loading a Tracker file opens all of the parts of a previous analysis. 


The main window displays the movie that is being analyzed as part of 
the experiment. The pane in the bottom right-hand side shows a table of 
x and y coordinates for a series of time units. A similar table is generated 
for each tracked object within the movie. In this example experiment, 
the only tracked object is the ball. The top right-hand pane contains the 
associated plot for the data stored in the table. This way, you can have a 
visual representation of the tracked motion. 

So, what can Tracker actually track? At the bottom of the movie display 
is a set of controls. If you click on the green arrow, the movie will start 
to play. As it does, Tracker actually registers the movement of the tracked 
object and also updates the data panes on the right-hand side. 

Although these panes do give you lots of raw data, there really isn’t any 
analysis being done yet. Clicking the View—Data Tool menu item pops 
open a new window where you can use this raw data to figure out what 
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Figure 4. 

As the movie 
progresses, 
Tracker updates 
the data panes 
on the right- 
hand side for 
the tracked 
object. 


Figure 5. 

The Data 

Tool window 
provides a set 
of analysis tools 
to look at the 
raw data pulled 
from the video. 


The Measure and Analyze buttons at the top of the plot window provide 
a set of different sections that you can display. This gives you several 
different analysis options for your data. Clicking the Measure button 
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Wi power [|cosine []sine 


Figure 6. Tracker also can do a Fourier analysis of the observed motion. 


gives you the options Coordinates, Slope and Area. When you select one 
or more of those options, values are displayed at the bottom of the plot 
representing each of the selected options. 

When you click the Analyze button, you get a drop-down with the 
Statistics, Curve Fits and Fourier Spectrum options. Each of those options 
actually opens a new pane in the Data Tool window. The exception is the 
Fourier Spectrum option. This option actually opens a new window where 
a Fourier analysis is done on the raw data from the tracked object. 

The Statistics option gives the data displayed in the top right pane. This 
is the set of descriptive statistics—that is, the maximum, minimum, mean 
and standard deviation, among other values. 

Selecting the Curve Fits option gives you the pane at the bottom of 
the plot window. Here you can select what type of curve is being fitted 
to your raw data. You can use a number of default functions to try to 
fit your data. In this case, the linear function is the obvious choice for 
a fitting function. You can see the function being used, x=A*t+B in 
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FitName:|Line 


Parameters = 


Name Expression 
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Figure 7. You define your own fitting functions as part of the data analysis. 


this case, along with the calculated values for the parameters A and 
B. If you are tracking an object that has a really odd motion, you can 
click the Fit Builder button to pop up a new window where you define 
a new fitting function. 

What do you do as part of a new project? Let's say you want to analyze 
the pendulum video from the sample video folder. You can set some 
calibration items within the video by either selecting the relevant buttons 
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Figure 8. You can set up calibration tools within your video to be analyzed. 


at the top of the window or selecting the items under the menu entry 
Track—New-—Calibration Tools. You can set up items like a calibration 
stick or a reference set of axes. 

One way to do the analysis is to step through the video, frame by 
frame, selecting the object to be tracked within each frame. This may 
be the only way to collect the raw data, but humans are lazy. If the 
object being tracked is relatively consistent across the duration of the 
video clip in question, you can try using the autotracker. Clicking the 
autotracker button will pop up a new window where you can control 
what is being tracked. 

Tracker helpfully displays reminder information on how to use the 
software. In this case, it reminds you that you need to press Ctrl, Shift 
and left-click the mouse button on the object of interest. If you then 
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= Fi 9. 

hab Autotracker: calibration stick A point 2 »* THe 
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Frame 0: No Template frame to frame 
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Template: Evolution Rate 20%|— Automark 4 

Search: |_| X-axis Only Look Ahead 


twee Tack Po 


To create a new key frame, shift-control-click the video feature of 
interest. Mouse over the controls above to learn more about settings 
and adjustments. 


| Show Key Frame | Delete | 


click the search button within the autotracker window, it will move 
through each frame, looking for the object in question. Once it is done, 
you should have a full set of data from the tracked object. From the 
plot in Figure 10, you can see right away that the pendulum follows the 
expected motion. You also can track multiple objects by creating a new 
track for each object. 
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Figure 10. The autotracker can generate all of your raw data automatically. 


Now that you've taken a quick look at Tracker and the kind of 
analysis you can do with it, you should be comfortable enough to 
start experimenting with it. Don’t forget to share any interesting 
ideas you come up with on the community page and add to a great 
resource for citizen science. 


—Joey Bernard 
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phone to it, but that’s already in your pocket anyway. 

The reason Google Cardboard can be so simple Is that its only job 
is to let you focus on a phone that is two inches from your face. 

A couple plastic lenses held in place by a chunk of cardboard isn't 
terribly high tech. But once you have that technology in place, 
Google Cardboard lets you literally transport to another world. Using 
apps from the Google Play Store, the VR experience is shockingly 
good using nothing more than a modern Android cell phone. 

The truth is, it takes a lot of graphical power to render two 
separate dynamic videos on a screen at one time. (That's how it 
makes the experience 3D.) So whatever VR app you download will 
likely have a slightly blocky-looking world. The crazy part Is, that 
doesn’t detract from the experience. In fact, some blocky trees almost 
make the experience that much more surreal because it’s obvious 
you're in a simulation. Yet, when you turn your head, it changes like 
you're actually there! 

If you want to experience some pretty cool virtual reality, but don’t 
want to invest in one of the expensive, Windows-centric systems 
like the HTC Vive, give Google Cardboard a try. You can get the 
actual cardboard goggles on Amazon, or you could spend a few 
more dollars to get a plastic set that Is a little more sturdy. Either 
way, the experience is amazing! It’s so amazing, in fact, that Google 
Cardboard gets the Editors’ Choice award this month. Virtual reality 
using the same technology that transports pizza? Yes please! 

—Shawn Powers 
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Testing 
Models REUVEN M. 


You've created a machine-learning model. Now, LERNER 
how do you know if it works well? 


Reuven M. Lerner offers 
training in Python, Git and 
PostgreSQL to companies 
around the world. He blogs 


NEXT at http://blog.lerner.co.il, 
oO PREVIOUS Dave Tayl or’s >) tweets at @reuvenmlerner 
Editors’ Choice and curates 
Work the Shell http://DailyTechVideo.com. 


Reuven lives in Modi’in, 
Israel, with his wife and 
three children. 


IN MY LAST FEW ARTICLES, I've been dipping into 
the waters of “machine learning”—a powerful idea 
that has been moving steadily into the mainstream of 
computing, and that has the potential to change lives 
in numerous ways. The goal of machine learning is 

to produce a “model”—a piece of software that can 
make predictions with new data based on what it has 
learned from old data. 

One common type of problem that machine learning 
can help solve is classification. Given some new data, 
how can you categorize it? For example, if you're a 
credit-card company, and you have data about a new 
purchase, does the purchase appear to be legitimate 
or fraudulent? The degree to which you can categorize 
a purchase accurately depends on the quality of your 
model. And, the quality of your model will generally 
depend on not only the algorithm you choose, but also 
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the quantity and quality of data you use to “train” that model. 

Implied in the above statement is that given the same input data, 
different algorithms can produce different results. For this reason, it’s not 
enough to choose a machine-learning algorithm. You also must test the 
resulting model and compare its quality against other models as well. 

So in this article, | explore the notion of testing models. | show how 
Python's scikit-learn package, which you can use to build and train 
models, also provides the ability to test them. | also describe how 
scikit-learn provides tools to compare model effectiveness. 


Testing Models 

What does it even mean to “test” a model? After all, if you have built 
a model based on available data, doesn’t it make sense that the model 
will work with future data? 

Perhaps, but you need to check, just to be sure. Perhaps the algorithm 
isn’t quite appropriate for the type of data you're examining, or perhaps 
there wasn’t enough data to train the model well. Or, perhaps the data 
was flawed and, thus, didn’t train the model effectively. 

But, one of the biggest problems with modeling is that of 
“overfitting”. Overfitting means that the model does a great job of 
describing the training data, but that it Is tied to the training data so 
closely and specifically, it cannot be generalized further. 

For example, let’s assume that a credit-card company wants to 
model fraud. You know that in a large number of cases, people use 
credit cards to buy expensive electronics. An overfit model wouldn't 
just give extra weight to someone buying expensive electronics in 
its determination of fraud; it might look at the exact price, location 
and type of electronics being bought. In other words, the model will 
precisely describe what has happened in the past, limiting its ability 
to generalize and predict the future. 

Imagine if you could read letters that were only from a font 
you had previously learned, and you can further understand the 
limitations of overfitting. 

How do you avoid overfit models? You check them with a variety 
of input data. If the model performs well with a number of different 
inputs, it should work well with a number of outputs. 
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In my last article, | continued to look at data from a semi-humorous 
study in which evaluations were made of burritos at a variety of 
restaurants in Southern California. Examining this data allowed one to 
identify which elements of a burrito were important (or not) in the overall 
burrito’s quality assessment. Here, in summary, are the steps | took inside 
a Jupyter notebook window in order to create and assess the data: 


%pylab inline 


import pandas as pd # load pandas with an alias 
from pandas import Series, DataFrame # load useful Pandas classes 
df = pd.read_csv('burrito.csv') # read into a data frame 


burrito_data = df[range(11, 24) ] 
burrito_data.drop(['Circum', 'Volume', 'Length'], axis=1, inplace=True) 
burrito_data.dropna(inplace=True, axis=0) 


burrito_data['overall' ] 


x < 
ool 


burrito_data.drop(['overall'], axis=1) 


from sklearn.neighbors import KNeighborsRegressor # import 

# classifier 
KNR = KNeighborsRegressor( ) # create a model 
KNR.fit(X, y) # train the model 


So, is the model good or not? You can know only if you try to make 
some predictions for which you know the answers, and see whether 
the model predicts things correctly. 

Where can you find data about which you already know the 
answers? In the input data, of course! You can ask the model (KNR) 
to make predictions about X and compare those with y. If the model 
were performing categorization, you even could examine it by hand 
to get a basic assessment. But using regression, or even a large-scale 
categorization model, you're going to need a more serious set of metrics. 

Fortunately, scikit-learn comes with a number of metrics you can use. If you say: 


from sklearn import metrics 
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then you have access to methods that can be used to compare your 
predicted values (that is, from the original “y” vector) to the values 
that were computed by the model. You can apply several scores to the 
model; one of them would be the “explained variance score”. You can 


get that as follows: 
y_test = KNR.predict (X) 


from sklearn import metrics 
metrics.mean_squared_error(y_test, y) 


Notice what's happening here. You’re reusing the input matrix X, 
asking the model to predict its outputs. But, you already know those 
outputs; those are in y. So now you see how closely the model comes 
to predicting outputs that already were fed into it. 

On my system, | get 0.64538408898281541. Ideally, with a perfect 
model, you would get a 1, which means that the model Is okay, but 
not amazing. 

However, at least you now have a way of evaluating the model and 
comparing it against other models that might be better or worse. You 
even can run KNR for different numbers of neighbors and see how well 
(or poorly) each model does: 


for k in range(1,10): 
print(k) 
KNR = KNeighborsRegressor(n_neighbors=k ) 
KNR.fit(X, y) 
y_test = KNR.predict (X) 
print "\t", metrics.mean_squared_error(y_test, y) 
print "\t", metrics.explained_variance_score(y_test, y) 


The good news is that you have now looked at how the KNR model 
changes when configured with different values of n_neighbors. 
Moreover, you see that when n_neighbors = 1, you get no error and 
100% explained variance. The model is a success! 
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Split Testing 

But wait. The above test is a bit silly. If you test the model using data that 
was part of the training, you would be surprised if the model didn’t get it 
at least partly right. The real test of a model is how well it works when it 
encounters new data. 

It’s a bit of a dilemma. You want to test the model with real-world 
data, but if you do that, you don’t necessarily know what answer should 
appear. And, that means you can’t really test it after all. 

The modeling world has a simple solution to this problem. Use only a 
subset of the training data to train the model, and use the rest for testing it. 

scikit-learn has functionality that supports this “train-test-split” 
functionality. You invoke the train_test_split function on your 
original X and y values, getting two X values (for training and testing) 
and two y values (for training and testing) back. As you might expect, 
you then can train the model with the X_train and y_train values and 
test it with X_test and y_test: 


from sklearn.cross_validation import train_test_split 
X_train, X_test, y_train, y_test = train_test_split(X, y, 
™test_size=0.25) 


KNR = KNeighborsRegressor(n_neighbors=1) 

KNR.fit(X_train, y_train) 

y_pred = KNR.predict(X_test) 

print "\t", metrics.mean_squared_error(y_test, y_pred) 

print "\t", metrics.explained_variance_score(y_test, y_pred) 


Suddenly, this amazing model no longer seems so amazing. By checking 
it against values it hadn’t seen before, it's giving a mean squared error of 
0.4 and an explained variance of 0.2. 

This doesn’t mean the model is terrible, but it does mean you might 
want to check it a bit further. Perhaps you should (again) check additional 
values of n_neighbors. Or, perhaps you should try something other than 
KNeighborsRegressor. Again though, the key takeaway is that you are 
now using a real, reasonable way to evaluate that model, rather than just 
eyeballing the numbers and assuming (hoping) that all is well. 
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What you really need to do is try 
different splits, so you can be sure that 
no matter what training data you use, 
the model performs optimally. 


Multiple Splits 
The split-test that you do might somehow tickle the model in such a 
way that it gives particularly good (or bad) results. What you really 
need to do is try different splits, so you can be sure that no matter 
what training data you use, the model performs optimally. Then, you 
can average the results over a bunch of different splits. 

In the world of scikit-learn, this is done using KFold. You indicate 
how many different instances of the model you'll want to create and 
the number of “folds” (that is, split tests) you'll want to run: 


from sklearn.cross_validation import KFold, cross_val_score 
kfold = KFold(n=len(X), n_folds=10) 


With the kfold object in place, you then can pass it to the 
cross_val_score method in the cross_validation module. 
You pass it the model (KNR, in this case), X, y and the kfold 
object you created: 


v_results = cross_val_score(KNR, X, y, cv=kfold) 

The cv_results object you get back describes the cross validation 
and typically is analyzed by looking at its mean (that is, what was the 
average score across those runs) and the standard deviation (that is, 


how much variance was there across runs): 


print cv_results.mean() 
print cv_results.std() 
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In this particular case, the results aren't that promising: 


0.310254620082 
0.278746712239 


In other words, although the n_neighbors=1 seemed to be so 
terrific when first analyzed, using all of the training data for testing, 
that no longer appears to be the case. 

Even if you stick with KNR as your classifier, you still can incorporate 
KFold, checking to see when (if) a different value of n_neighbors 
might be better than the value of 1 you gave here: 


from sklearn.cross_validation import KFold, cross_val_score 


for k in range(1,10): 
print(k) 
KNR = KNeighborsRegressor(n_neighbors=k ) 
kfold = KFold(n=len(X), n_folds=10) 
cv_results = cross_val_score(KNR, X, y, cv=kfold) 
print "\t", cv_results.mean() 
print "\t", cv_results.std() 


Sure enough, when k=9, you get results that are significantly better 
than when k=1: 


0.594573190846 
0.161443573949 


That said, | do believe it’s likely you can create a better model. 
Perhaps a better classifier for regression would improve things. 
Perhaps using categorization, rather than regression, in which you 
round the values in y to the nearest integer and treat scores as 
5 distinct categories, would work. Perhaps, as mentioned before, 
| should have paid more attention to which columns were most 
(and least) important and done some better feature selection. 

Regardless, with a proper test system in place, you’re now able 
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to start tackling these questions intelligently with a way to evaluate 
your progress. 


Summary 

It's not enough to create a machine-learning model; testing it is also 
important. As you saw here, scikit-learn makes it relatively easy to 
create, split-test and then evaluate one model or even a whole bunch 
of them. 

Supervised learning isn’t the only type of machine learning out 
there. In many cases, you can ask the computer to divide your data 
into multiple groups based on heuristics it develops, rather than 
categories that you have trained. In my next article, | plan to look at 
how (and when) to build “unsupervised learning” models. m 
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Is the Moon 
Waxing or 
Waning? 


Dave finalizes his phase of the moon script. 


PREVIOUS NEXT 
Reuven M. Lerner’s Kyle Rankin’s >) 
At the Forge Hack and / 


I DON’T KNOW ABOUT YOU, but I’m still thinking 
extra-planetary thoughts as we go through the tail 
end of this particularly contentious election season 
and its aftermath. Maybe life on other planets is 
easier? Ah, maybe not. 

In any case, | completed the Martian lander and 
now am enmeshed in a phase of the moon script. 
In my last article, | talked about the complications 
of calculating the phase of the moon and decided 
simply to scrape the same web site that Google 
uses: http://www.moongiant.com. 

That site provides the current moon illumination 
level, which lets you break it down into the phases 
of new moon, crescent, quarter, gibbous and 
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Therefore, to ascertain waxing or waning, all 
you need to do is know the moon’s illumination 
level today and either yesterday or tomorrow. 


full. Amateur astronomers know that the fun part of tracking the 
moon's phase is to understand whether it’s “waxing” (growing more 
illuminated) or “waning” (growing less illuminated). 

Although at any given moment the moon is illuminated based on 
its location, and your location, relative to the sun, the full cycle of a 
moon phase starts and ends with a new (0% illuminated) moon, and 
the full moon (100% illuminated) is the mid-point of the journey. 

Therefore, to ascertain waxing or waning, all you need to do is 
know the moon's illumination level today and either yesterday or 
tomorrow. Fortunately, the Moon Giant website obligingly has the 
ability for you to ascertain the illumination level for a specific date. 

A quick visit to the site with a regular web browser reveals that it works 
using a date-based URL format like this: http://www.moongiant.com/ 
phase/11/6/2016. 

So, you can build the date URL for the day before today with a call 
to the date program. If you've got the GNU version of date, it’s easy 
to back up a day: 


$ date 

Mon Nov 7 11:40:31 MST 2016 
$ date -v -1d 

Sun Nov’ 6 11:40:15 MST 2016 


It turns out that you also can specify that you want to back up 24 
hours, although, of course, the net result is the same: 


$ date -v -24H 
Sun Nov 6 11:40:24 MST 2016 
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More important, you can pass date a format string that you then can 
evaluate with the eval function, so you can set month, day and year for 
yesterday in one easy step: 


$ eval $( date -v -1d +"mon=%m day=%d year=%Y" ) 
$ echo month = $mon, day = $day and year $year 
month = 11, day = 06 and year 2016 


It’s quite a handy trick when you need to work with extracting specific 
elements from date and 10x that when it also involves date math. 


Older Date Programs Are More Complicated 

But, what if your version of date doesn’t include the -v flag and 
doesn’t have all these fancy features? Then, my friend, you are 
facing a definite challenge. Date math is pretty easy, except for 
the edge cases. 

That is, it’s easy to extract the current month, day and year from 
even the most rudimentary Linux version of date, and it’s obviously 
easy to subtract one from the day, but what if it’s the first of the 
month? Or the first of the year? 

That’s doable too, but it’s just a bit more work. Notably, you'll also 
want to know about leap years, because one day prior to March 1, 
2016, might be February 28, or it might be February 29, depending 
on whether 2016 was a leap year. 

Now a sneaky way to do it simply would be to sidestep the issue. If 
the day number of the month is greater than 1, subtract one to get 
yesterday's date. If it is the first, however, add one and reverse the 
logic of the waxing/waning test. 

Fortunately, | do have the more sophisticated date program, so I’m 
going to do that most frustrating of things and leave this particular 
facet as the proverbial exercise for the reader. 


Yesterday’s Lunar [Illumination Level 

Knowing the format of the Moon Giant URL when you specify a 
date, and knowing how to use eval and date to get the numeric 
month, day and year values for yesterday, here’s some code to put 
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But wait, “waxing” and “waning” applies only 
to crescent and gibbous moon phases. If the 
moon is new, quarter or full, neither word 
applies in common astronomical parlance. 


that all together: 


url_ago="http://www.moongiant.com/phase" 
eval $( date -v -1d +"mon=%m day=%d year=%Y" ) 
ydayurl="$url_ago/$mon/$day/$year" 


The good news is that the resultant web page has the same format 
as the page for today’s illumination level too, so the same curl|grep 


sequence explored in my last article can be reused for this task: 


yillumlevel="$( curl -s "$ydayurl" | grep "$pattern" | tr ',' '\ 
' | grep "$pattern" | sed 's/[40-9]//g')" 


In fact, let's add a debugging statement that displays both today’s lunar 
illumination level and yesterday’s level: 


echo today\'s illumination level = $illumlevel and \ 
yesterday was $yillumlevel 


Running it on November 7, 2016, here’s what the script and the Moon 
Giant website report: 


today's illumination level = 47 and yesterday was 37 
Now it’s a simple test: is today’s level greater or less than 
yesterday's level? 


But wait, “waxing” and “waning” applies only to crescent and 
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gibbous moon phases. If the moon is new, quarter or full, neither 
word applies in common astronomical parlance. 
Seriously, who came up with these rules? Talk about complicated! 
Here’s how this all fits together 


if [ $illumlevel -gt $yillumlevel ] ; then 
# we're waxing if it's getting brighter 
waxwane="Waxing" 

else 
waxwane="Waning" 

ie 

if [ $illumlevel -1lt 5 ] ; then 
phasename="new" 

elif [ $illumlevel -1lt 45 |] ; then 
phasename="$waxwane crescent" 

elif [ $illumlevel -1lt 55 |] ; then 
phasename="quarter" 

elif [ $illumlevel -1t 95 |] ; then 
phasename="$waxwane gibbous" 

else 
phasename="full" 

ie 

echo "The moon is currently $phasename with \ 

$illumlevel% illuminated." 


That's just about the entire script. If | run it on November 7, 2016, the 
moon is 47% illuminated, which makes it a quarter moon (45%-55%), 
so the output Is: 

The moon is currently quarter with 47% illuminated. 
A few days later, on November 10, the output is what you would hope: 


The moon is currently waxing gibbous with 78% illuminated. 


Done. Nice and easy. 
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Creeping Featurism 

You could do plenty of things with this script to improve it and make it 
more powerful and flexible. The easiest would be simply to rewrite that 
output line so it’s less grammatically awkward: 


echo "It's a $phasename moon that's $illumlevel% illuminated." 


Now the output will make a bit more sense as the script reports 
that “It’s a waning gibbous moon that’s 72% illuminated.” 

The bigger task is to allow users to specify a date and calculate the 
values for that particular date (including the day prior to the date 
specified). | would do this using the same basic date -v approach, 
but first parse users’ input and break it down into month, day and 
year. For simplicity’s sake, constrain their input to a MM/DD/YYYY 
format, and there’s surprisingly little involved in the task. 

For huge bonus points, of course, a graphical display would be 
nice. But that’s hard to do with a shell script, right? 


Next Month 

That's it for space. Next month, I’m planning to turn back to games 
and explore how to write a rock, paper, scissors game. You might 
want to study the game first so you're ready! m 
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IN MY LAST ARTICLE, | introduced how MCollective 
could be used for general orchestration tasks. 
Configuration management like Puppet and Chef can 
help you bootstrap a server from scratch and push 

new versions of configuration files, but normally, 
configuration management scripts run at particular 
times in no particular order. Orchestration comes 

in when you need to perform some kind of task, 
specifically something like a software upgrade, ina 
certain order and stop the upgrade if there’s some kind 


52 | January 2017 | http://www.linuxjournal.com 


HACK AND / 


of problem. With orchestration software like MCollective, Ansible or even 
an SSH for loop, you can launch commands from a central location and 
have them run on specific sets of servers. 

Although | favor MCollective because of its improved security model 
compared to the alternatives and its integration with Puppet, everything 
| discuss here should be something you can adapt to any decent 
orchestration tool. 

So in this article, | expand on the previous one on MCollective and 
describe how you can use it to stage all of the commands you'd normally 
run by hand to deploy an internal software update to an application server. 

| ended part one on MCollective with describing how you could use it to 
push an OpenSSL update to your environment and then restart nginx: 


mco package openssl update 
mco service nginx restart 


In this example, | ran the commands against every server in my 
environment; however, you'd probably want to use some kind of 
MCollective filter to restart nginx on only part of your infrastructure at a 
time. In my case, I’ve created a custom Puppet fact called hagroup and 
divided my servers into three different groups labeled a, b and c, split 
along fault-tolerance lines. With that custom fact in place, | can restart 
nginx on only one group of servers at a time: 


mco service nginx restart -W hagroup=c 


This approach is very useful for deploying OpenSSL updates, but 
hopefully those occur only a few times a year if you are lucky. What 
you more likely will run into as a common task ideal for orchestration is 
deploying your own in-house software to application servers. Although 
everyone does this in a slightly different way, the following pattern is 
pretty common. This pattern is based on the assumption that you have a 
redundant, fault-tolerant application and can take any individual server 
offline for software updates. This means you use some kind of load 
balancer that checks the health of your application servers and moves 
unhealthy servers out of rotation. In this kind of environment, a simple, 
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serial approach to updates might look something like this: 
M@ Get a list of all of the servers running the application. 
m Start with the first server on the list. 
m Set a short maintenance window for the server in your monitoring system. 
m@ Tell your load balancers to drain any existing sessions to this server. 
m Update the list of available packages for the server. 
m Stop the service on that server. 
m Update the software on that server. 
m Start the service on that server. 
m Make sure the service started successfully. 
m Perform a health check to make sure the service is healthy. 
m Add the server back to the load balancer rotation. 
mM Repeat for the rest of the servers on the list. 

If any of those steps fails, the administrator would stop the update 
and go investigate and fix the problem. Often if there is going to be 
a failure, it will be at the software-update or health-check phase, and 
the point of this process is to make sure that if an upgrade doesn’t go 
well, you stop at the first server before pushing broken software to the 
rest of the environment. 

Traditionally, administrators might perform all of the above steps 
manually by logging in to different servers and interacting with 


different web interfaces perhaps. The next step they follow generally 
involves wrapping a series of SSH commands that would perform these 
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actions into a shell script and then maintain some local configuration 
file that defines lists of servers. 

With MCollective, the process is similar with the main difference 
being that MCollective doesn’t need to have SSH root privileges on 
these machines. Instead, MCollective performs its tasks by putting a 
limited set of commands in a job queue that all of the servers check. 
The commands are restricted by what MCollective plugins you have 
installed on a particular server, and MCollective does a good job of 
sanitizing input from the plugins it includes by default. 

Most of the above commands in that deploy list can be completed 
using the default plugins MCollective includes. | use Nagios for 
monitoring, and although MCollective does include a plugin that 
lets you perform NRPE commands (a Nagios agent that runs on each 
server that allows Nagios to run local commands to check disk space, 
RAM and so on), it doesn’t include anything that could directly set a 
maintenance mode in Nagios. 

Another missing piece in the above list of commands is the ability 
to interact with a load balancer. Many people might skip this step 
these days, as they are using something like nginx’s internal load- 
balancing abilities and may not have an easy way to set something 
like a maintenance mode to drain existing connections to a host. In 
that case, you may just skip ahead to stopping the service and let the 
health check detect the failure. That approach risks dropping existing 
connections though, and because | use Haproxy as my load balancer, 
| can use its built-in command mode to set a maintenance mode on 
specific servers if I'm logged in to the load balancer. 

Fortunately, MCollective has the ability to extend its existing set 
of commands with your own custom plugins to perform specific 
tasks. Unfortunately, writing, packaging and deploying even trivial 
MCollective plugins can be a bit complicated the first time you do it, 
and it’s involved enough that it would require an article all of its own. 
McCollective’s plugin documentation is a good place to start, and in 
particular, the documentation on writing plugins that use MCollective’s 
RPC framework makes the code you have to write much more 
straightforward, even if you aren’t familiar with Ruby. 

When you write a custom MCollective plugin, you choose a new plugin 
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name (say, haproxy) and then define a list of commands you want to 
pass that new plugin (such as disable_server and enable_server). 
If acommand needs some kind of argument passed to it, you also 
define those. Then, you map those commands and arguments into basic 
command-line commands using their RPC framework, or you can dig in 
to using native Ruby libraries if you are familiar with that. 

| wrote a custom Nagios plugin and an Haproxy plugin that would 
send my custom commands to their command file and command 
socket, respectively. So to set a maintenance mode on server1.example.com 
for Nagios and Haproxy, | would type these commands: 


mco rpc nagios maintenance server=serveri1.example.com duration=5m 
mco rpc haproxy disable_server server="Serverrole/server1i" 


Because | took advantage of MCollective’s RPC framework, | have to 
type rpc in front of my custom commands. 

Next | provide the name of my plugin, then the command | want to 
run, followed by any custom arguments. Then on the Nagios server 
side, | intercept that command and format it into a format | can write 
to Nagios’ local command file so it can execute. In the case of the 
Haproxy plugin, this command goes out to any server that happens 
to be running Haproxy. If a particular Haproxy server doesn’t have my 
server defined in its configuration, it doesn’t do anything harmful, and 
otherwise, it sets it to maintenance mode. 

With these plugins in place, you can replace the above generic list of 
steps to specific MCollective commands: 


M mco find -S "domain=example.com and 
resource(’Package[myapp]).managed=true" 


M mco rpc nagios maintenance server=myapp1.example.com 
duration=5m 


M mco rpc haproxy disable_server server="myapp/myapp1" 


M mco rpc package apt_update -I myapp1.example.com 
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M mco service myapp stop -I myapp1.example.com 

M mco package myapp update -I -I myappi.example.com 
M mco service myapp start -I myapp1.example.com 

M mco service myapp status -I myapp1.example.com 

M mco nrpe check_app_health -I myapp1.example.com 

M mco rpc haproxy enable_server server="myapp/myapp1" 


I've ended up wrapping all of these commands inside a basic shell 
script that takes the name of a particular application as an argument, 
then performs the first mco find command to get the list of servers 
that have that package installed. Then at that point, | just run the next 
set of commands in a basic for loop. Where appropriate, | added a 
Sleep command here and there to give a service time to come up. If 
any of the commands fail, the script exits out and reports the error 
so the administrator can investigate. Otherwise, it runs through each 
server in order. 

Of course, later versions of this script have become a bit more 
sophisticated, so it can accept some custom arguments, log the 
output to a known log file and be more efficient in how it sleeps 
between commands. But the end result for the sysadmin is a simple 
“deployapp” script they can run that properly updates the application 
the right way, every time, with no risk of skipping or forgetting a 
server or steps in the process.m 
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My Love 
Affair with 
Synology 


What has 48TB and is full of penguins? 


oO eae NEXT ’b) 
Hack and / New Products 


IN MY RECENT "HODGE PODGE” ARTICLE (in the 
October 2016 issue), | mentioned how much | love 
the Synology NAS | have in my server closet (Figure 
1). | got quite a few email messages from people— 
some wanting more information, some scolding 
me for not rolling my own NAS, and some asking 
me what on earth | need with that much storage. 
Oddly, the Linux-running Synology NAS has become 
one of my main server machines, and it does far 
more than just store data. Because so many people 
wanted more information, | figured I'd share some 
of the cool things | do with my Synology. 
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Figure 1. The Synology DS1815+ is what | use, but the entire line of Synology NAS 
devices shares a common interface. 


Why So Much Storage?! 
| guess | should address the reason | have 48TB (36 usable) of storage 
(Figure 2). | store a lot of data (har har har). Seriously though, | have a 
local copy of close to 100,000 photos, 1000s of hours of home videos 
and several complete Linux distribution repositories. That takes a lot 
of storage! The bulk of my needs, however, comes from entertainment 
media. Ever since my kids first used DVDs to skate across the kitchen 
floor, I've been backing up my movies digitally to my server. Through the 
years, that has migrated from DVD ripping to Blu-ray ripping, but years of 
movies really add up. Even those aren't the bulk of my data, however. 

| collect television series. Sometimes those collections are ripped from 
my TiVo, manually edited and converted to MKV. If I'm being honest, 
however, most of my television shows are just downloaded from torrent 
sites. Yes, | know it’s not kosher to download torrents of television shows. 
But | also know that | pay more than $200/month to the cable company 
for every channel available, and if | wanted to take the time, | could do 
the TiVo rip/edit/convert dance. | just don’t have the time. Because | pay 
for cable access, it doesn’t bother me to download television shows. (We 
actually do buy all our Blu-ray movies though. I’m not a proponent of 
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Figure 2. The dashboard shows you information on your NAS at a glance. I’m slowly 
building my collection after the horrible data loss | suffered a few years ago. 


pirating things you don’t have rights to.) It's okay if you disagree with my 
choice to download television shows via torrents, | get it. Really, | do. Just 
ignore those parts of this article! 


What Kind of Drives? 
Don’t skimp on hard drives. That’s generally good advice regardless of the 
situation, but with NAS devices, please spend the extra money to get drives 
rated for NAS. | have eight 6TB Western Digital Red NAS drives. When | 
bought them, the WD Red Pro drives weren’t available. Still, the standard 
Red drives are rated for up to eight drive bays, so I’m still within spec. 

| haven't always been so picky about drives. In fact, | just used to 
get the biggest, cheapest drives | could. Since | use RAID6, a drive or 
two failing isn’t a big deal—except that | actually had three drives fail 
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at exactly the same time, and | lost all my data, including family home 
movies that | didn’t have backed up anywhere. It still hurts. So really, 
don’t skimp on drives, it’s just not worth it. (Also remember to back up, 
even large files. RAID isn’t a backup, trust me.) 


Why Synology? 
I've had Drobos, QNAPs and multiple Netgear devices. They all sucked. 
No, really. The performance on every single device I’ve had in the past 
has been horrible (even with good drives), and I’ve never been able to 
determine exactly why. Once more than one simultaneous read happens 
over the network, they all just crap out. With the Synology, | can have 
four 1080p video streams going at once without any slowdown at all. 
The other thing | like about the Synology is its software. Most other 
NAS devices have apps that you can install on the Linux system, but the 
Synology apps seem to be more elegant and work reliably (Figure 3). In 
fact, there are some incredible things | do with the NAS device that I’m 
sure weren't exactly what it was designed to do (more about that in a bit). 
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Figure 3. The apps are plentiful, and there are community-supported unofficial apps as well. 
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Ultimately, the biggest draw for me is how well Synology keeps itself 
updated and maintains its drives. It automatically does scans and integrity 
checks, plus it does system updates without disrupting the servers | have 
connected to it via NFS. Every other NAS I’ve used stays at whatever 
software version it comes with, because upgrading the firmware almost 
always means drive failures and server lockups. I’m sure there are 
procedures for QNAP and such that make upgrading possible, but the 
Synology does it automatically—and | like that a lot. 


TV and Torrents 

| like the SickRage program not only because it automatically searches 
and downloads new episodes of my television shows, but also because 

it organizes my existing collection. | have every episode of Star Trek that 
ever has been produced (including the animated series from the 1970s), 
and SickRage does an incredible job of naming and organizing those files. 
As long as | spent ripping the Star Trek the Next Generation DVDs, | don’t 
ever want to have to figure out which episode is which again! 

In order to install SickRage, you actually need to install “Sick Beard 
Custom” and then paste in the SickRage Git URL. The short version of 
the story is that Sick Beard was the original program, but the developer 
stopped developing it, so folks forked it, and SickRage is the best fork out 
there, by far. Even if you’re not using Synology, you should be running 
SickRage. Head over to https://github.com/SickRage/SickRage for the repo 
or http://sickrage.github.io for the home page. 

SickRage supports lots of torrent clients, and it supports NZB too. 
I've found NZB to be less reliable than it used to be, so I’ve moved 
back to 100% torrents. | like the Transmission web interface, so that’s 
what | use on Synology. It’s another maintained app, so just search 
for “transmission” in the package installer application. Integrating 
Transmission and SickRage is beyond the scope of this article, but 
rest assured, it’s not difficult. SickRage is designed to work with 
Transmission, so setting it up is easy. Warning: if you use SickRage 
and Transmission to download television shows, you will get DMCA 
take-down notices from your ISP. Apparently the production companies 
disagree with my rationale for downloading TV episodes. Thankfully, 
| have a solution for that. 
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Networking and Traffic Routing 

My Synology device has four Gigabit Ethernet ports. | think that’s overkill, 
but since the software allows me to bond the four ports together (even 
with a switch that doesn’t support 802.3ad), I’m happy to have more 
bandwidth than | need. | never have an issue with throughput, even when 
streaming those multiple video files mentioned above. 

Since Synology supports VPN connections, the first thing | did was set 
up my privateinternetaccess.com account so my torrents would be directed 
through the VPN. | haven't gotten port forwarding to work through the 
VPN, but even without a redirected port, my torrents download fine. The 
problem is my VPN connection occasionally goes down. When it does, the 
torrents go through my gateway, and even when the VPN comes back up, 
the tracker connects me via the non-VPN connection. And, | get DMCA 
notices. This is very frustrating. So | decided to remove the gateway device 
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Figure 4. Notice the gateway is in the 10.x.x.x range, which is not what | use on my local 
network. That is assigned by the VPN. 
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from the Synology altogether! Bear with me. 

| have a network address assigned on my local network so LAN computers 
can connect. That works fine. Without a gateway specified, however, the 
NAS can’t connect to the internet for torrents, SickRage or even system 
updates. But when the VPN is connected, it sets the gateway address 
automatically to an address on the other side of the VPN (Figure 4). As long 
as my VPN is connected, the system has a gateway assigned, and it can 
access everything through the VPN. If the VPN goes down briefly, rather 
than defaulting to the local network gateway, it just can’t connect to the 
internet. Once the VPN is re-established, it reassigns a VPN gateway, and 
boom, the NAS ts back online! The only problem is how can | connect to the 
VPN if | can’t get on the internet? The answer: static routes. 

If you look at Figure 5, you'll see that | have a static route set up so 
that traffic going to the IP address of my VPN goes through my LAN’s 
gateway. Since it’s only a static route for that network, the rest of the 
internet is still inaccessible. | also could do fancy firewall work and allow 
the NAS to access only the VPN and drop all other packets, but | like 
the solution to be self-contained. That way, if | change routers or router 
configs, | don’t have to worry about getting DMCA notices. 
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Figure 5. This is the sneaky static route so | can connect to my VPN, but nothing else. 
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The Synology also will act as a router, forwarding traffic. That means 
| can point my Roku to the Synology as its gateway device, and I'm 
able to watch local blackout games on the MLB.tv app, because all the 
traffic goes through the VPN. The only change | have to make is on 
my DHCP server, which gives the Synology’s IP address as the Roku’s 
gateway address. It works perfectly and saves me setting up another 
VPN to get around MLB's regional restrictions. (Honestly, | usually 
watch baseball games on TiVo, but occasionally the game is on only via 
streaming, and | like having that option.) 


Backups 

Remember when | said RAID wasn’t a backup? Yeah, | meant that. I’ve 
lost too much valuable data through the years to depend on RAID to 
protect my files—even when the drives and NAS device seem to be 
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Figure 6. Backup solutions are in great supply. 
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more solid than any I’ve had in the past. Thankfully, Synology has a 
few different backup options (Figure 6). The most practical one for 
large amounts of data is the Hyper Backup app. It has the ability to 
copy your entire NAS to a variety of destinations. Whether you choose 
to buy another Synology NAS and store it in your shed or back up your 
data to Amazon Glacier, the same Hyper Backup program can handle 
the regular updates. 

| don’t want to pay for Amazon Storage, even though the Amazon 
Drive Unlimited is decently priced at $60/year. | worry that my 30TB 
would cause Amazon to invent a reason to suspend my account. Plus, 
it would take so long to back up my entire data store to the cloud, 
that it literally might never get done. Right now, | just back up my 
irreplaceable files (home movies, photos and so on). Someday | hope to 
get a second Synology NAS and set up that “mirror in the shed”. Still, 
Synology has so many backup options, it’s hard to find a reason to 
delay setting up a backup solution! 


Things I Don't Do 
The Synology had a decent processor, and the RAM is even 
upgradeable. Still, it’s not a beefy server when it comes to resource- 
hungry applications. For example, even though the Plex Media Server 
is available in the package management system, I’d never install it. 
Plex uses way too much CPU to transcode video streams. I’m thankful 
the Synology is powerful enough to stream the actual video files over 
fileshares, but the thought of transcoding 1080p MKV streams in real 
time? It’s a bad idea. | have a standalone server | use for Plex Media 
Server, and while it can transcode at least four full resolution video 
streams, it’s also a huge i7 CPU with a boatload of RAM. Unless you're 
doing minimal streaming with low-resolution video, | encourage you to 
avoid Plex Media Server on any NAS device. 

| also can’t run the really amazing reverse proxy server on Synology. 
The setup is easy, and the configuration is very intuitive, but my 
VPN/no-gateway setup means that the reverse proxy doesn’t work 
outside my network. Even if | forward a port to the NAS from my 
router, it tries to send responses out the VPN connection and fails. 
Reverse proxies are easy enough to configure on any other machine 
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in my network, so it’s not a huge loss, but it’s worth noting that it’s 
something my crazy VPN system breaks. 


Not the Only Option 

Before you think | was paid by the folks at Synology to brag about 
their product, | will freely admit that a big tower server with a bunch 
of hard drives and software RAID makes for an incredible NAS. It 
means you can beef up the hardware too and do things like run 

Plex Media Server. | simply like the efficiency of the Synology devices. 
They're fast, cool running and just sip electricity. I'm sure there are 
other brands of NASes that do a decent job too, and Synology isn’t 
perfect. In all honesty, however, it’s the best product I’ve been able 
to find, and | have literal piles of junk NAS devices that just couldn't 
do the job. If you're looking for a NAS device, in my opinion, you 
can’t go wrong with Synology. 
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Rogue Wave Software's TotalView 
for HPC and CodeDynamics 


New versions of not just one but two dynamic analysis tools 
from Rogue Wave Software were unveiled recently to pleased 
developers everywhere. Upgraded TotalView for HPC and 
CodeDynamics, versions 2016.07, improve the diagnosis and 
correction of bugs, memory issues and crashes at execution. The 
heart of these tools’ innovation is support for 64-bit ARM v8-A 
architecture. ARM's presence continues to expand the ecosystem 
for the HPC space and enables highly integrated, energy-efficient 
solutions. With TotalView for HPC and CodeDynamics, customers 
using ARM platforms have a robust, scalable dynamic analysis 
solution for their complex HPC clusters and codes. Also included 
are overall enhancements, platform updates and improved 
support for operating systems and compilers. 
http://roguewave.com 
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Panther MPC, Inc.'s Panther Alpha 


Functional, powerful and there when you need it, unobtrusive when 
you don’t—that’s how Panther MPC, Inc., sums up the company’s 
new Panther Alpha personal micro PC that features the company’s 
powerful, easy-to-use Linux-based Panther OS. Panther Alpha 
combines full desktop functionality with an ultra-customizable Linux 
OS that fits in the palm of your hand. Panther says its new device 
could be possible only now thanks to a culmination of years of 
industry innovation and development, namely the power of today’s 
ARM chips and an improved emphasis of Linux on ARM. The Panther 
Alpha is whisper quiet, with no fan or HDD noises, and is powered by 
an energy-efficient ARM quad-core chipset. Unlike most micro PCs, 
asserts Panther, the Panther Alpha offers a true desktop experience 
with mouse, keyboard input, microSD card slot, Ethernet, Wi-Fi, 
Bluetooth and USB functionality. The portable and compact shape 
allows it to integrate seamlessly into any environment—business, 
home or school. “The only PC you'll ever need”, adds the company. 
http://panther-mpc.com 
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PrestaShop 


Helping people overcome the challenges of building and growing 
an online business is what the PrestaShop open-source ecommerce 
platform is all about. The significant PrestaShop 1.7 release provides 
innovations focused on three themes: sell faster, create easier and 
code better. PrestaShop 1.7 users will sell faster due to the entirely 
redesigned UX that helps merchants with the daily management 

of their stores via more effective back-office management and a 
drastically reduced time to place products online. PrestaShop 1.7 
users will create easier with new features like the starter theme, a 
simple and flexible tool with all the functionality needed to design 
the perfect store. The starter theme gives web designers significant 
opportunity for creativity and customization. Finally, PrestaShop 
1.7 users will code better with the incorporation of new tools and 
standards, including the popular Symfony framework and PHP 7 
compatibility. The goal is to improve site performance and code 
security as well as to facilitate better development. PrestaShop 

1.7 is already available in 25 languages and through community 
crowdsourcing, it gradually will be available in many more. 
http://prestashop.com 
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SoftMaker’s FlexiPDF 


Editing PDFs is now as easy as word processing. This is SoftMaker’s 
promise thanks to its new FlexiPDF 2017, a new PDF editor that 
“masters the creation of new PDF files as well as the editing 

of text, graphics and drawings in existing ones”. Available in 
Standard and Professional versions, FlexiPDF goes far beyond 
basic editing functions, empowering users to change almost any 
aspect of PDF files. Just like in standard word processors, FlexiPDF 
2017 offers an integrated track changes function, built-in spell 
checker, commenting function, search and replace, highlighting 
and exporting of PDFs to TextMaker, HTML, RTF, Microsoft Word 
and EPUB formats. The integrated OCR function in FlexiPDF 
converts scans into editable PDF files. FlexiPDF comes with its 
own PDF printer driver that lets users create high-quality PDFs 
directly from within any Windows application that can print. 
FlexiPDF Professional is perfect for the translation of PDF files 
into other languages. This is because it extracts the text of a PDF 
document in a format that can be opened by common translation 
programs. It then imports the translation back to the original PDF 
document—in exactly the right spot. 

http://flexipdf.com 
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Goldtouch Semi-Vertical Mouse 


“A brilliant combination of ergonomics, comfort and 
performance” thunders Goldtouch, a leader in desktop 
ergonomics, about its new Semi-Vertical Mouse. “Perfectly sloped 
at 66 degrees” to prevent wrist stress and provide the ultimate 
comfort fit for healthy computing, the Semi-Vertical Mouse is 
Goldtouch’s newest addition to its mouse library. The new mouse 
provides a happy medium between the traditional ergonomic 
mouse (sloped at 25-30°) and the traditional vertical mouse 
(sloped between 85-90°). The in-between slope guides users 

into a comfortable grip, provides a nice resting position for the 
hand and enhances the user experience through great mousing 
precision. Goldtouch says that the Semi-Vertical Mouse expands 
its already wide variety of ergonomic solutions to fit every type 
of user and user preference. 

http://goldtouch.com 
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3 ») 
= = Pretty much anything in the 
O'Reilly spin-off Make: series is like 
catnip to us Linux cats, and the 
cordon an new book Making Things Smart is 
we. no exception. The book is subtitled 
Easy Embedded ARM Programming 
For Transforming Everyday Objects Into Intelligent Machines and 
is authored by Gordon H. Williams. The book is Williams’ tool 
for teaching readers the fundamentals of the powerful ARM 
microcontroller by walking beginners and experienced users 
alike through easily assembled projects composed of inexpensive, 
hardware-store parts. In rebellion to the many current ARM 
programming books that take a bland, textbook-ish approach with 
focus on complex, beginner-unfriendly languages—think C or ARM 
Assembler—Making Things Smart utilizes Espruino (JavaScript for 
Hardware) to flatten the learning curve. 
http://oreilly.com 
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Buddy Platform Limited’s Parse on 
Buddy Service 


With Facebook’s Parse mobile back end as a service shutting 
down in January 2017, developers are in a bind. The vise is 
squeezing tighter since the open-source Parse Server product 
released by Facebook, asserts Buddy Platform Limited, was not 
designed to support high volume, commercial-grade apps from 
organizations seeking the breadth of the original platform. To 
solve this issue and facilitate an easy migration path to a robust 
alternative, Buddy launched an enterprise-grade Parse service 
offering called Parse on Buddy. Parse on Buddy’s developer says 
that its solution is closest to the original Parse service, including 
many features not part of the Parse Server codebase released to 
the Open Source community (for example, cloud code, multiple 
apps per account support, PPNS, file storage and more). Parse on 
Buddy was designed to be as familiar to developers as possible, 
right down to using the same pricing model. It also features an 
easy three-step migration process for apps still on the original 
Parse service that requires only a small change to a mobile app’s 
codebase and no remapping of APIs or other new code. 
http://buddy.com 
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Clarity 


VMware's Clarity Design System 


By combining user experience (UX) guidelines and patterns 
with the front-end code in one solution, VMware's Clarity 
Design System represents a new concept in the design systems 
space. Clarity Design System is VMware's open-source project— 
“designed by designers and built by developers”—that unites 
UX, HTML/CSS and Angular for creating cohesive experiences 
for any application. VMware began building Clarity based 

on readily apparent needs for an internal design system 

that unified VMware's portfolio of products and accelerated 
product development through reusable components. Building 
alongside product teams, the Clarity team was able to tackle the 
complexities of enterprise use cases while striving for simplicity 
familiar to consumer applications. Now dozens of VMware 
product teams have adopted Clarity internally in their work. 
http://vmware.github.io/clarity 


Please send information about 
releases of Linux-related products 
to newproducts@linuxjournal.com 
or New Products c/o Linux Journal, 
PO Box 980985, Houston, TX 77098. 
Submissions are edited for length 
and content. 
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Password managers make it easy to have unique user 
names, unique email addresses and unique passwords for 
each account. They also provide a secure store for 
extra account details. Additional features like notes and 
attachments allow you to use password managers as 
secure escrow files like digital safety deposit boxes. 
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nternet and cloud companies do not have perfect security. You're all 
familiar with large-scale data theft from big corporations. Although 
most enterprises do a great job considering the attacks against 
them, you can do your part by protecting your own accounts. 

It’s common wisdom to use a different password for each account. 
Better yet is using a unique email address as well. Doing so quickly 
becomes unwieldy, however, so you find you need an external brain. 

Password managers function well as this external brain and can help with 
more than just passwords. They make it easy to have unique entries for user 
names, email addresses, security-question answers and much more. 


Database Entries Groups View Tools’ Help 


tal) ke S| a & 
Root > LinuxjJournal > Edit entry 


Title: Linuxjournal 


Advanced 
icon Username: der.hans 
Auto-lype Password: 
; * SSOSSSOSSSOSSOSOSSSOSOSSOSOSSOSSSSSD 3 
Properties 
History Riemer YI TIT IIiiiiiiiiiiiiiiiiiiiiiiiy | Gen. 


URL:  https://www.Linuxjournal.com/ 


[Expires 30/16/21 1-46 pm Presets v 


Notes: 


@ OK X& Cancel 


Figure 1. Password managers should have fields for record name, user name, website 
and notes. 
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Rather than creating a 

whole new email account, 
you can likely take advantage 
of sub-addressing. 


Password managers securely encrypt data before storing it. In addition 
to passwords, they should have fields for record name, user name, 
website and notes. 

The following example uses the apg (Automated Password Generator) 
command to create a random string of text: 


$ apg -n1 -Mnl 
vucapob7 


Now you have a unique user name for your bank. The next social- 
media site thieves will have to figure out both your user name and your 
password to attack your bank account. 

Use a unique email address. Rather than creating a whole new email 
account, you can likely take advantage of sub-addressing. 

Some email providers have sub-addressing, which allows a separator 
character and then a token. When delivered, the email provider 
ignores the separator character and the text after it. For example, 
yout+vucapob/7-mybank@example.com would be delivered to 
you@example.com. See the Email Sub-addressing sidebar for more information. 

Unique email addresses make it harder for thieves to social-engineer 
companies, as they won't have your user name, whether or not it’s an 
email address. 

Since you're storing credentials in a password manager, you don’t need 
to memorize or even see them. In fact, now you've turned your password 
manager into a credential manager holding unique passwords, unique 
user names and unique email addresses. Just a few paragraphs in, and 
you're already powering up! 


78 | January 2017 | http://www.linuxjournal.com 


FEATURE: Online Privacy and Security Using a Password Manager 


CREATING GOOD PASSWORDS 


When using a credential manager, you need to memorize only a few 
passwords—for instance, your password to authenticate to your 
computer and credential manager, and both for work as well. 

Here are some tips on building strong, but memorizable passwords: 


1) Use multiple unrelated words, at least four. 


2) Include numbers, capital letters (not just at the beginning of a word) 
and punctuation. Perhaps use them rather than spaces. 


3) If fluent in multiple languages, use them. 

4) Use nonsensical, uncommon words—no common phrases. For 
instance, start with four unrelated words. This is probably fine for 
computer and network logins that change every few months: 


anteater twiddle aquamarine drove 


5) Use more than four words for your password manager where backups 
might be captured for long-term, offline attacks: 


ameisenbaer twiddle aguamarina drove preamble esac 

6) Even better, add some non-space transitions: 

aMeisenbaer%tWiddle aGuamarina9dRovezpReamblezeSac 

On a mobile device, that last example is a pain. Especially when using a 
simpler password, minimize the number of entries in the mobile password 


database, and change anything in it on a regular basis in case some app 
ships the file off to a botnet for cracking or the phone is stolen. 
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One key to having 
unique security questions 
is to lie. 


But, as the infomercials say, there’s more. 

The notes field allows you to store multi-factor emergency codes, shoe 
size and anything else that might be useful for the account. You could 
store customer-service phone numbers, cheat codes for getting to an 
actual person and notes from the last call. 

Some sites have undocumented password requirements, such as not 
allowing % in a password. The notes field is a good place for a reminder 
for you. If you use NoScript and a cookie blocker, you also might want to 
add notes about required JavaScript and cookie domains to use the site. 

One key to having unique security questions is to lie. The company 
doesn't need to know your grandma’s favorite first high school mascot. It 
just needs a response only you can give. Random text to the rescue: 


$ apg -n1 -m15 -Mnl 
icunyedgicekoco 


That’s a fun one, but it'll be challenging if you have to say it 
over the phone. apg can help with that. The -t option says to give 
pronunciation guides: 


$ apg -n1 -m1i5 -Mnl -t 
icunyedgicekoco (ic-un-yed-gic-ek-oc-o) 


It still sounds like a foreign language, but it’s hardly guessable. 
The credential manager doesn’t care—it’s just data. 


KeePassX Credential Manager 
| recommend using KeePassX combined with KeePassDroid. This article 
uses KeePassX 2.x. Most of it applies also to the KeePassX 1.x versions, 
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Database Entries Groups View’ Tools’ Help 
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Welcome! 


Figure 2. KeePassX Opening Screen 


but there are some differences. 

Select Database—New Database to create a new credential database. 

This password you have to memorize. If you forget the password for 
your KeePass file, you can’t open it. See the Creating Good Passwords 
sidebar for how to choose wisely. 

You also can use a key file. The key file holds the key used to unlock the 
database, and your password opens the key file. An advantage is that the 
keyfile can be stored separately from the credential database. 

Once you've entered your password, | recommend you save the file and 
close it, then open the file again to make sure you can type in your new 
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Database Entries Groups View Tools’ Help 


ma ip eis 
Change master key 


(@) Password 
Enter password: jee@eggececeoooooooooocoeeoeoeeeeeeeeee 


Repeat Password: @@@00000000000000000000000000000000000008 


[_} Key file 


@ OK X Cancel 


Figure 3. Creating a New Database 


password. Open and close it three or four times to help you memorize 
the password. After 15 minutes, do the close and open dance again, then 
make sure you test it the next morning as well. Forgetting that password 
is the same as losing the file—the data is unavailable. 

To add a new entry, either click the key icon with the green arrow or 
the Entries+Add New Entry drop-down menu. The drop-down shows a 
third option, the Ctrl-n shortcut. 

KeePassxX can autogenerate passwords. In addition to specifying 
character groups like upper and lowercase letters, numbers and special 
characters, you can choose length and exclude look-alike characters, such 
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Database Entries Groups View Tools’ Help 


a «» a & 
Root > Add entry 


Title: 


Advanced 
Icon Username: 
Auto-lype Password: 
Properties 
History Repeat: Gen. 
PaSSWOrd: P@SSCCOOCOOOOCOOCOOOOOOOOOOOOOOOCE 
Length: i a PT Fey ices, 42 5 
Character Types 
A-Z a-z 0-9 nearer 
(lM) Exclude look-alike characters 
(@| Ensure that the password contains characters from every group 
Accept 
URL: 
(_] Expires Presets v 
Notes: 
@ OK & Cancel 


Figure 4. Autogenerating a Password 


as O and O and 1 andl. 

Choose a long length like 42 characters since you're not memorizing 
the passwords. The eye button will let you view the random text if you 
need to (which is useful if a site discourages good passwords by blocking 


password pasting). Click the accept button to get KeePassX to store the 
new entry, then save the file. 
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In addition to the notes field, KeePassX 2.x has fields for additional 
attributes. The latter is a better place for storing security questions and 
answers. Although both fields display the values in plain text, the default 
screen does not show additional attributes, and there’s a drop-down 
menu to use them. 

Additional attributes can be found in the Advanced section for an 
entry. To copy them, use the Entries drop-down, select Copy Attribute 
to Clipboard, then select the attribute you want. 

For securely copying a secret, it’s important to keep it hidden and 
keep it safe. The password stays hidden because it isn’t displayed. 

It’s kept safe because it expires out of the clipboard. By default, the 
clipboard is cleared after ten seconds. 


Database Entries Groups View Tools’ Help 


i “> = & 
Application Settings 
General (®) Clear clipboard after 19 sec A 
: 


[| Lock databases after inactivity of 


[_} Show passwords in cleartext by default 


(| Always ask before performing auto-type 


@ OK X Cancel 
Figure 5. Clear Clipboard Setting 
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KeePassX provides handy keyboard shortcuts. Ctrl-b copies the user 
name into the clipboard for pasting, and Ctrl-c copies the password. 
Review the drop-down menus to find more shortcuts. 


Credential Manager Features 
You should require the following features in a credential manager: 


M Clipboard clearing. 

m Password generation. 

mM Pronounceable password option. 

mM Encrypted data with operating-system-independent storage. 
m Backups and data liberation. 


The clipboard should be cleared automagically after a short time. There's 
no need for it to stick around and get pasted somewhere accidentally. 

A credential manager should have a password generator. Even better is 
a random text generator available from anywhere in the application that 
simplifies creating random values for security questions and responses, 
sub-addressing tokens and birthdates. 

The random text generator should have an option for pronounceable 
passwords. KeePassX 1.x does, but KeePassX 2.x does not. Hopefully the 
feature will be added back in. apg’s pronunciation guide is quite useful. 
Neither version of KeePassX has that. 

The data file must be encrypted and interoperable. In order to be 
trusted, the encryption software and algorithm need to be free and 
open so they can be audited independently. KeePassX is GPLv2 or later, 
and KeePassDroid is GPLv3. Both incorporate code under other free 
software licenses. 

Interoperability means usable by multiple applications. KeePassX and 
KeePassDroid use free and open KeePass file formats. KeePass, kpcli 
and other applications also can use the files. None can open the file 
without the master password. They are your passwords; you should 
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have exclusive access. 

Being operating-system-independent is important for secure data. 
KeePassX runs on GNU/Linux, BSD, Mac and Windows. KeePassDroid will 
pout both user name and password into the notification drawer for easy 
selection on the phone. 

You need to have backups and data liberation. Since KeePass files are 
encrypted, copying them elsewhere is sufficient for backups. Keep old 
master passwords secure as they can be used on old backups of the file. 

Data liberation means you can access your data even if the original 
application or service provider stops working. A standard format like 
KeePass works since many applications can read it. Additionally, KeePassXx 
exports to a plain-text file. Make sure to protect it with an encryption 
tool, such as GnuPG. 

For online password managers, data liberation means you can get 
your data back out of the service provider in an open format. If you use 
a proprietary service, you are dependent on that service allowing you 
access. Make regular exports to an open format in case the provider goes 
offline, decides you haven't paid recently or jumbles your data. 

| recommend separate files for personal and work. No need for a 
subpoena about some work contract to have access to all of your personal 
credentials. Just like email, keep personal and work in separate accounts. 

For traveling or mobile use, you don’t need all of your credentials. You 
can drop the whole file on some file-sharing service and unlock the entire 
file from your phone, but then a compromise on your phone has access to 
all of your accounts as well. 

Keep an updated file with a subset on the mobile device. | recommend 
the minimum essentials to connect to your file-sharing service and 
operate if you have voice, but not data. 

KeePassX does not provide a way to sync data to another location 
securely, but kpcli can export a group to a new KeePass 1.x file. For 
instance, create a “mobile” group in your KeePass file, then use kpcli to 
export that group. kpcli is available under the same licensing as Perl. 


Other Useful Features 
A credential manager also can be used to escrow data like family social 
security numbers, important private dates, insurance account information 
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and anything else your family might need should you no longer be 
available. KeePassX also can store attachments. Now you have a data 
escrow manager. Power up. 

A credential manager should liberally allow text for the master 
password. KeepPass files do. As expected, KeePass accepts special 
characters. In fact, when | give presentations on KeePassxX, | often 
demo unlocking a KeePass file with cut and paste of Perl and shell 
Script snippets. 

KeePassxX has an Auto-Type feature that will log you in to 
a website with one hot key: Ctrl-v. Although other people 
like Auto-Type features, | find it too easy to mis-click and 
paste my credentials in the wrong place. Eventually | do get it 
right, usually long before “usert+twyimCij5-fsf@example.com 
ZFSZ5ZWEhSFHRGYf474MCRQ8pW4YNDXT87wrsQhkyL” starts 
trending in Google searches. 

KeePass files also support an expiration date for passwords, and 
KeePassX provides preset time periods like three and six months. Once 
the entry expires, KeePassX marks it when viewing the group as a 
reminder to change the password. 

KeePass allows you to access previous versions of an entry via its history 
feature. It’s not quite revision control, but it’s still handy. 

It’s also handy to provide a means of live security and authenticity testing. 

For instance, LastPass is an online commercial password manager 
with a history of quickly responding to needs. For example, shortly after 
Heartbleed was announced, LastPass integrated a check to verify the 
destination website did not have the Heartbleed bug before sending 
credentials. It also does some verification to make sure you're connecting 
to the actual site rather than some phishing site. 

LastPass Is mostly browser-based. The company also provides a 
command-line tool with source available, and a perusal of several files 
yielded GPLv2 or later licensing. 

LastPass does encryption and password creation on the client side, 
so the company never has access to your unencrypted data. It shares 
credentials by encrypting on the client side with the recipient’s public key. 
Again, LastPass doesn’t see unencrypted credentials. 

LastPass also advertises a feature that allows the other person to use 
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the credentials without seeing them. Although LastPass can’t view the 
unencrypted data, it’s fairly easy for the recipient to capture the hidden 
plain-text password. 

See the Password Managers sidebar for a list of free software 
password managers. 

If you prefer password generation and password storage to come from 
different applications, the Generating Random Text sidebar has some 
ideas for creating your own random strings.m™ 


der.hans is a free software, technology and entrepreneurial veteran. His roles have included director of 
engineering, engineering manager, IS manager, community college instructor, developer, DBA and his 
favorite, system administrator. He is also now a repeat author for Linux Journal. In his free time, der.hans 
endeavors to help build the Free Software community through user group and conference leadership. 
He is chairman of the Phoenix Linux User Group (PLUG), BoF organizer for the Southern California Linux 
Expo (SCaLE) and founder of the Free Software Stammtisch. He's currently supprting manufacturing in 
the US as a senior engineer at Shutterfly. 


RESOURCES 


KeePassX Licensing: https://github.com/keepassx/keepassx/blob/master/COPYING 
KeePassDroid Licensing: https://f-droid.org/repository/browse/?fdid=com.android.keepass 
Automated Password Generator (apg): http://www.adel.nursat.kz/apg 

LastPass: https:/Awww.LastPass.com 

Sub-addressing RFC: https://tools.ietf.org/html/rfc5233 


LJ article on KeePassX 1.x—“KeePassX: Keeping Your Passwords Safe” by Anthony Dean: 
https://www.linuxjournal.com/content/keepassx-keeping-your-passwords-safe 


Mat Honan: https://Awww.wired.com/2012/08/apple-amazon-mat-honan-hacking 


Send comments or feedback via 
http://www.linuxjournal.com/contact 


or to ljeditor@linuxjournal.com. 


RETURN TO CONTENTS 
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GENERATING RANDOM TEXT 


The biggest key to a strong password is entropy, randomness. Use long, pure 
random text to get high entropy. A credential manager isn’t the only way to create 
long strings of random text. For instance, let your cat walk on your keyboard. 


Here are some methods that are more efficient and less scratchy than trying 
to train your cat to work on demand: 


1) Use a random text generation tool, such as apg, pwgen or makepasswd. 


2) Use GnuPG’s random function: 


$ gpg --gen-random --armor 1 42 


3) Use OpenSSL’s random function: 


$ openssl rand -base64 42 


4) Use uuidcdef to create a UUID: 


$ uuidcdef -u 


5) Create a hash of a real answer plus a master password as the salt: 


$ echo "anteater twiddle aquamarine drove \ 
LinuxJournal.com" | sha256sum | cut -c 42 


A feature of this method is that one master password can derive any site 
password without saving anything to disk. There’s even a tool to help with that: 


$ passwordmaker -r LinuxJournal.com -g 42 -a SHA1 \ 
-x -m "anteater twiddle aquamarine drove" 


One problem with derive-on-the-fly is that you don’t have per-site random text. 
passwordmaker does allow using an unencrypted configuration file. 


6) Use date to generate a random birthdate: 


$ date -d @$( (SRANDOM*24*3600/2-500000000)) +%Y%b%d 
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EMAIL SUB-ADDRESSING 


Sub-addressing augments your email address by inserting tokens. It turns 
one email address into as many as you want. 


A separator character is placed after the user name portion of an email address, 
and then you can add a token, <username><character><token>@<domain>. 
You decide on the token. 


Tokens can protect your accounts as listed in the article and also filter 
email. For instance, phishers like to send fake eBay mail. It’s not eBay’s 
fault, but it’s a popular target. With an eBay token, you know that email 
without the token, user+pl8qr-ebay@example.com for instance, isn’t from 
eBay and can be trashed automagically. 


Some common email providers and their separator characters are: 
@ Gmail: + 

M@ FastMail.FM: + 

M Kolab: + 


M Yahoo!: - 


Gmail sub-addressing can be disabled at the domain level, so it might not 
be available for work accounts. 


FastMail.FM also offers sub-domain as an alternative to sub-addressing. 
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EMAIL SUB-ADDRESSING 
ON YOUR OWN SMTP SERVER 


Some common SMTP servers and their default separator characters are: 
M@ Postfix: + 
mM Exim: default off 


M Courier and qmail: - 


Some considerations if adding sub-addressing to a domain: many 
websites disallow a plus sign in an email address; periods are allowed 
everywhere, but often they already are in use for first.last@domain; dash 
is also fairly universally allowed. 


PASSWORD MANAGERS 


Many free software password managers are available: 
KeePassx: https://www.keepassx.org 

KeePassDroid: http://www.keepassdroid.com 

kpcli: http://kpcli.sourceforge.net 

KeePass: http://www.keepass.info 

Password Safe: https://pwsafe.org 

pass: https://www.passwordstore.org 


Console Password Manager (cpm): http://github.com/comotion/cpm 


pwman3: https://owman3.github.io/pwman3 


KDE and GNOME have built-in password managers, and there are web 
browser plugins. 
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n my previous two articles (see the November and December 2016 
issues of LJ), | described setting up a 6LOWPAN network and then 
integrating that into the internet so that the low power sensor/ 
actuator nodes can talk to internet hosts and vice versa. This Is 
one of the major mechanisms currently proposed for bringing the 
Internet of Things (loT) to life. 

Once you have established a communications pathway, however, you 
need to look at how you are going to use that pathway to exchange 
information—specifically, the protocols and the data types. The 
currently favored protocols are MQTT (MQ Telemetry Transport) and 
CoAP (Constrained Application Protocol), and they each fill different 
roles. MQTT is a messaging system using publish/subscribe, which has 
been adapted for low power devices. CoAP is similar to, and based 
on, HTTP but is heavily optimized for low power devices. This article 
focuses on CoAP. 


CoAP 

The World Wide Web is built on the HTTP protocol. This is a traditional 
client/server model, where clients connect to a server over TCP and make 
requests of the server, which in turn prepares replies and delivers them to 
the client. The outstanding success of the Web has led to this being used 
as the model for CoAP, with the following appropriate changes: 


™ HTTP is TCP-based. A TCP session requires a handshake setup, 
acknowledgment of packets, retries on failure and keepalive 
mechanisms. UDP is much lighter; packets are sent via a “send and 
forget” mechanism. CoAP uses UDP, and it is up to the sender as to 
whether it requires an acknowledgement. 


m Because HTTP is a transport protocol, any application-layer protocol can 
be built on top of it. One of the worst of these was SOAP, a protocol 
that has had some success in enterprise systems. CoAP uses REST, 
which is much closer aligned with HTTP. 


® Application data traditionally has been attached to HTTP packets using 
formats like XML and, more recently, JSON. These are text-based, 
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and consequently, they are heavy both in payload and in processing. 
CoAP applications can use any of these, but the trend is toward CBOR 
(Concise Binary Object Representation), a binary version of JSON. 


m HTTP does not allow messages to be sent from the server to clients, 
so COAP has added a mechanism for this. This is coming into HTTP 
through “server push” mechanisms. 


REST 

REST (REpresentational State Transfer) is the philosophy behind HTTP, 
described by Roy Fielding (the principal HTTP 1.1 architect) in his PhD 
thesis. In a horribly emasculated form, he says that 1) resources are 
identified by URIs, such as Web URLs, and 2) resources are accessed 
using only four verbs: GET, PUT, POST and DELETE, with defined 
meanings (although when to use PUT and when to use POST is still 
debated). These definitions are: 


™ GET: get a representation of a resource. For sensor data, this most 
likely will be in JSON or CBOR format, and will contain data, such as 
the temperature of a sensor. 


m PUT: set a new value for a resource. For a heating system, it could be 
setting a new temperature value. 


m POST: usually used to create a new resource, and possibly of limited use 
for sensors and actuators. 


m DELETE: delete a resource. Again, it’s possibly of limited value here. 
| provide a more expansive version at https://jan.newmarch.name/ 
loT/Middleware/REST. 


CoAP and Python 

CoAP will most likely be run as a server on sensors and actuators. These 
won't be highly endowed with RAM, and they are actually unlikely to be 
able to run Python, which takes megabytes of RAM. Even micro-Python 
takes about 180kB of RAM. Most likely, they will run compiled code, using 
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a library such as the libcoap C library (https://libcoap.net). 

I'm running these examples on Raspberry Pis, so I’m using Python 
for simplicity. There are many Python packages for CoAP and many 
implementations of CoAP for other languages (see http://coap.technology/ 
impls.html). The Ubuntu x86 repositories have the aiocoap Python 
package, so you can install that on your desktop with: 


sudo apt-get install python3-aiocoap 


The RPi repositories currently have no CoAP packages, so you will 
have to install something (again!). You need to get the CoAP package 
on the sensor RPi. Download it from aiocoap, the Python CoAP library 
(https://github.com/chrysn/aiocoap). It contains the Python libraries in the 
alocoap directory as Python code. You can move that directory to, say, 
/usr/lib/python3.4, so that it can be found from any Python 3 program: 


git clone --depth=1 https://github.com/chrysn/aiocoap.git 
cd aiocoap/ 
sudo mv aiocoap /usr/1lib/python3.4 


The package also contains clientGET.py, clientPUT.py and server.py. 
These not only demonstrate the CoAP package, but they also test some 
features. (I'll adapt these to our purpose here.) 


A Simple CoAP Application 
I'm going to use the CPU temperature example from my previous two 
articles, as it is about as simple as one can get. 

The sensor has to be exposed as a resource—that is, have a URI (here a 
URL). This will use the scheme coap:// or coaps://, its IPv6 address and 
its URI path, such as temperature. Note that the sensor will be running 
as a server—the client will be making queries to the server. 

To the client, the URL will look like this: 


coap://[fd28::2]/temperature 


using the global IPv6 address you set on the “sensor” RPi in the 
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previous article. The IPv6 address needs to be in square brackets ([...]) 
to avoid the colons (:) being confused with a URL Port option. The 
default UDP port is 5683. 

The alocoap package uses the recently added yield from Python 
generator system. | won't go into that here (it is non-trivial). The major 
parts to note are what you configure in the client and server. 

The client needs to set the method as GET to fetch the CPU temperature 
of the server, using the server’s URI. Then it reads a response and does 
something to it. Here you just print the response. The client is: 


#!/usr/bin/env python3 
import asyncio 
from aiocoap import * 


@asyncio.coroutine 
def main(): 
protocol = yield from Context.create_client_context() 


request = Message(code=GET ) 
request.set_request_uri('coap://[fd28::2]/temperature' ) 


try: 

response = yield from protocol.request(request).response 
except Exception as e: 

print('Failed to fetch resource:') 

print(e) 
else: 

print('Result: %s\n%r'%( response.code, 

response.payload.decode( 'utf-8'))) 


if __name__ == "__main__": 
asyncio.get_event_loop().run_until_complete(main() ) 


The server uses the asynchronous I/O package asyncio. Again, you can 
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ignore the details of this. The important thing is to add resources that can 
be accessed by CoAP user agents (clients). You add a new resource with: 


root.add_resource(('temperature',), TemperatureResource() ) 


which sets the URI-Path (/temperature) of the resource on this host and 
a class (TemperatureResource) to be invoked when the resource is 
requested. Any number of resources can be added, such as pressure, 
motion and so on, each with their own class handler. 

The handling class is the most complex, and there are many possibilities. 
The simplest will subclass from aiocoap.resource.Resource 
and will have a method render_get, which is called when a GET for 
a representation of the resource is needed. For the example sensor, 
this gets the CPU temperature as before and then wraps it into an 
aiocoap.Message. Here's the server code: 


#!/usr/bin/env python3 


import asyncio 
import aiocoap.resource as resource 
import aiocoap 


from subprocess import PIPE, Popen 


class TemperatureResource(resource.Resource): 
der .. init. .(Sselr ): 
super(TemperatureResource, self).__init__() 
@asyncio.coroutine 
def render_get(self, request): 
process = Popen(['vcgencmd', 'measure_temp'], stdout=PIPE) 
output, _error = process.communicate( ) 
return aiocoap.Message(code=aiocoap.CONTENT, payload=output ) 


def main(): 
# Resource tree creation 
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root = resource.Site() 

root.add_resource(('temperature',), TemperatureResource( ) ) 
asyncio.async(aiocoap.Context.create_server_context(root) ) 
asyncio.get_event_loop().run_forever() 


if __name__ == "__main__": 
main( ) 


The output from running the client against this server is similar to this: 


Result: 2.05 Content 
"temp=36.9'C\n" 


as In previous examples. 


Making Things Reusable 
What I've basically done at this point is hack up an example to show 
how CoAP works, but the loT isn’t going to succeed if programmers 
act like that. My sensor will need to work in your environment, talking 
to other people’s systems. The loT isn’t going to be a simple monolithic 
environment. It’s going to be a mess of multiple systems trying to talk 
to each other. 

Standards and conventions will need to be agreed upon, and not 
just between people, but in ways that can be read and confirmed by 
machines. I’ve used CoAP over 6LOWPAN, and that is just one battle that 
is raging. The next one is over data formats and device descriptions—both 
using them and discovering them. 


Data Formats 

HTTP has mechanisms to query and to specify data formats. For HTTP, this 
is managed by Content Negotiation, and this idea is carried across into 
CoAP: a client can request particular data formats, while the server may 
have preferred and default formats. 
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XML is generally regarded as too heavyweight. JSON is better, but 
as a text format, it still carries baggage. CBOR (Concise Binary Object 
Representation) is an IETF RFC for a binary encoding of JSON and is 
becoming popular. It has an advantage of being self-contained, unlike 
other recent binary formats, such as Google’s Protocol Buffers, which 
require an external specification of the data. 

A JSON format of the sensor data could look like this: 


t 


"temperature" : 37.9, 
*UnLtS”- = "GC" 


CBOR translates this into a binary format, which may be more concise. 
To use CBOR, first you need to install it. Python packages normally 
are installed using pip, and the RPi does not come with this installed. 
So install both it and the cbor module (note that you want the Python 

3 versions): 


sudo apt-get install python3-pip 
sudo pip3 install cbor 
sudo pip3 install LinkHeader 


Then, a JSON equivalent data type can be encoded using cbor.dumps, 
which creates a byte array and is decoded by cbor.loads, which turns it 
back into a Python type. A Python dictionary is equivalent to the JSON of 
a JavaScript class object given above. 

The server is modified by code to create a Python dictionary and then 
turn it into CBOR. The client is likewise modified to decode the CBOR 
data into a Python dictionary. You also will do some elementary content 
specification, using I[ANA-registered numbers. The application/cbor 
number is 60, from the IETF RFC 7049. 

The relevant part of the server is this: 


CONTENT_FORMAT_CBOR = 60 
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class TemperatureResource(resource.Resource): 
def __init__(self): 
super(TemperatureResource, self).__init__() 


@asyncio.coroutine 
def render_get(self, request): 
process = Popen(['vcgencmd', 'measure_temp']|, stdout=PIPE) 
output, _error = process.communicate( ) 
list = re.split("[='\n]", output.decode('utf-8')) 
dict = {'temperature' : float(list[1]), ‘unit' : list[2]} 
mesg = aiocoap.Message(code=aiocoap.CONTENT, 
payload=cbor .dumps(dict) ) 
mesg.opt.content_format = CONTENT_FORMAT_CBOR 
return mesg 


And, here's the relevant part of the client: 


request = Message(code=GET ) 
request.set_request_uri('coap://[fd28::2]/temperature' ) 


cry: 
response = yield from protocol.request(request).response 
except Exception as e: 
print('Failed to fetch resource: ') 
print(e) 
else: 
if response.opt.content_format == CONTENT_FORMAT_CBOR: 
print('Result: %s\n%r'%(response.code, 
cbor.loads(response. payload) ) ) 
else: 
print('Unknown format' ) 


This prints something like this: 


Result: 2.05 Content 
{'temperature': 37.4, ‘unit': 'C'} 
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Device Descriptions 

The code above is fine for interacting with a temperature sensor—once 
you know what that is! You may have hundreds of sensors of different 
types, and all you may know Is their IPv6 address. To complete this, you 
need to know the following: 


m What is the specification of a device, such as a “temperature sensor” ? 


m What are the special values for your sensor (for example, max 
and min temperatures)? 


m How do you tell what type of device you have? 
™ How do you know how the CoAP requests interact with your device? 


At the moment, there are no industry-agreed-upon answers to those questions. 
One could say that (unfortunately) this is another of the differentiators in the 
loT world. The IETF in RFC 7252 and RFC 6690 has made some progress, but 
there are still open issues, and they are not uniformly adopted. 

From RFC 6690, each device should have a URI-path of /.well-known/core, 
which can be accessed by an HTTP GET coap://<IPv6-addr>/ 
.well-known/core request. RFC 6690 specifies that the representation 
must be in CoRE Link Format, which | will describe soon. 

Two new link attributes are added to the standard Web link headers of 
RFC 5988, such as title. The new attributes are the following: 


m rt for resource type. 
m if for interface type. 


The values of these attributes can be strings, URLs or anything— 
this isn’t specified. The resource type is expected to be some 
“well known” value that identifies the type of device, such as 
jan.newmarch:temperature-sensor. Yes, | just made that 
up—there are several proposals but no standards yet. 

The value of if is supposed to be some specification of the REST 
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interface for the device—that is, how to call it using GET, PUT and so on, 
and what is returned from those calls. How an interface is described isn’t 
specified by RFC 6690. Although possibly using WADL (Web Application 
Description Language) is suggested, the Open Connectivity Foundation 
uses RAML (RESTful API Modeling Language), and the Wikipedia page on 
RESTful APls lists a dozen more, probably used by some group or other. 

Investigating REST API languages is beyond the scope of this article, so 
let’s just assume the well known core resource has a value like this: 


</temperature>; rt="jan.newmarch: temperature-sensor"; 
if="https://jan.newmarch.name/temperature-sensor" 


Here /temperature is the relative URL of the resource, the value of rt is 
the “well known” device type, and the value of if is the description of the 
device. Assume that https://jan.newmarch.name/temperature-sensor contains 
WADL or RAML or some other description that allows you to deduce that 
requesting the resource /temperature using GET will return a CBOR object 
with fields temperature and unit, with float and string values, respectively. 

The format of the well-known resource is defined to be in 
application/link-format, which according to the [ANA CoAP 
Content-Formats site (https://www.iana.org/assignments/core-parameters/ 
core-parameters.xhtml#content-formats) has CoAP code 40. The format is 
actually just UTF-8. 

The server is modified by adding another resource: 


root.add_resource(('.well-known', 'core'), WkKCResource(root) ) 


where WKCResource Is a Class in the aiocoap module, which keeps a list 
of all the resources supplied by this device. 

When the client GETs the resource /.well-known/core, it will get 
a comma-separated list like this: 


</.well-known/core>; ct=40, 

</temperature>; 
if="https://jan.newmarch/temperature-sensor"; 
rt="jan.newmarch.name: temperature-sensor" 
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For each resource, a client should extract the rt value. If it recognizes 
it as a temperature device, then it should carry on. If it doesn’t, it should 
look up the if URL and extract what the GET method can do, and then 
carry on. That code is not covered here. The CoRE Link Format string can 
be parsed using the Python LinkHeader package. 


Conclusion 
This series has addressed the issues of setting up a 6LOWPAN low power 
wireless network, using the OpenLabs radios on Raspberry Pis, followed 
by bringing these devices into internet visibility. This concluding article 
looks at data formats and protocols for the IoT. 

Many topics have been omitted. The major one is that of security, 
as the system | have described here is wide open to snooping and 
hacking. The security mechanisms are all there, but they are a full topic 
in their own right. 

| also have ignored the issue of how external clients find the IP 
addresses of the clients. This is answered by internet draft “CoRE Resource 
Directory draft-ietf-core-resource-directory” (https://tools.ietf.org/html/ 
draft-ietf-core-resource-directory-09). 

| haven't addressed networking within a 6LOWPAN network. There 
are a variety of models, such as mesh networking, and they build on 
the IEEE802.15.4 networking model. 

Finally, | haven't mentioned other pieces of hardware dealing with 
IEEE802.15.4 and 6LOWPAN. These include modules from Texas 
Instruments, Firefly and Libelium, with many others coming along.™ 


Jan Newmarch has been using Linux since kernel 0.96. He has written many books and papers about 
software engineering, network programming, user interfaces and artificial intelligence, and he is 
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ou had to be a crank to insist on being 


right. Being right was largely a matter of 
explanations. Intellectual man had become 
an explaining creature. Fathers to children, wives to 
husbands, lecturers to listeners, experts to laymen, 
colleagues to colleagues, doctors to patients, 
man to his own soul, explained. The roots of this, 
the causes of the other, the source of events, 
the history, the structure, the reasons why. For 
the most part, in one ear out the other. The soul 
wanted what it wanted. It had its own natural 
knowledge. It sat unhappily on superstructures of 
explanation, poor bird, not knowing which way to 
fly.—Saul Bellow, Mr. Sammler’s Planet, 1969. 
| began writing this column on November 9, 
2016, on the balcony of a hotel in Istanbul, while 
a call to prayer echoed through the streets below. 
| took that as good advice, because a few hours 
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earlier my country elected an Internet troll, Donald Trump, as its 
president (http://scripting.com/2016/07/28/dontFeedDjTrump.html). 
Perhaps by now we’re calling this day 11/9, in the mold of 9/11. I'm 
an optimistic guy, but color me pessimistic about where my country Is 
now heading, led by a world-class narcissist. 

And forgive me for obsessing not only about where this is going, 
but how we got here. Our country has been hacked, and that matters. 

Disclosure: I’m a political independent, and not a fan of Hillary 
Clinton, though | thought she was the only sensible choice, given 
Trump's shortcomings, many of which should have disqualified him, 
flat out. But he won. Why? 

| don’t know, though | did see it coming. Mostly | felt it. Polls said 
one thing, my senses another. “We know more than we can tell”, 
says Michael Polanyi. Evidence: most of the time we don’t know how 
we'll end the sentences we start, or how we started the sentences 
we end. Yet we know what we’re talking about. And if we succeed, 
another human being gathers our meaning, even though they can't 
repeat it verbatim. 

To say something Is to express some care about it. We also tend 
to hear what we like to hear more than what we don't, even if 
we welcome what might disagree with us. Those of us who work 
with logic (such as Linux Journal readers) have a high regard for 
the rational. But while logic and reason sit on the mental board of 
directors, emotions cast the deciding votes. As Bellow says, the soul 
wants what it wants. 

To see how emotions might cast deciding votes, Heartbeat Al studied 
emotional leanings in five “swing” states: ones a candidate needs to 
win in a close election (https://heartbeatai.com/#welcome-1). Before 
Election Day, ordinary polls showed Clinton winning most or all those 
states. But the Heartbeat Al study showed something very different. 
Here it is: https://go.heartbeatal.com/report/client/529-241866ea3095 
8b0b2007ffde94ddbed9. Play with it a bit. The little heart on the right 
is a tab that pulls out a drawer of variables you can turn on and off. 

| just did that, and Figure 1 shows both candidates’ sentiment 
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Figure 1. Hillary Clinton and Donald Trumps’ Sentiment Maps for Five “Swing” States, 


via Heartbeat Al 


maps, together. 


Trump won all those states. Hell, somebody had to. But the main point 
here is that voters disliked both candidates—a lot. They simply disliked 
Trump less than they disliked Clinton—not that they actually liked Trump. 

This kind of study doesn’t show a mandate, but it does suggest caution 
before suggesting that a victory by either party constituted a mandate of 
any kind—or should, anyway. 

It also makes one wonder how voters came to feel the way they did 
about the candidates. To what degree are those feelings attached to 
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actual facts? Hard to tell, but wondering should be productive. 
On September 18, 2016, | blogged this (http://doc.blog/2016/09/18/ 
trumpTheMule.html): 


As soon as it became clear that Trump was a breed apart, remarkable 
more for his powers of persuasion and enlistment than for anything 
else (his policies are all feints: magical misdirections away from his 
absolute vanity), | saw him as the Mule (https://en.wikipedia.org/wiki/ 
Mule_%28Foundation%29), star of Isaac Azimov's Foundation series. 
Here’s how Wikipedia describes the Mule: 


“One of the greatest conquerors the galaxy has ever seen, he is a 
mentalic (https://en.wikipedia.org/wiki/Mentalic) who has the ability to 
reach into the minds of others and ‘adjust’ their emotions, individually 
or en masse, using this capability to conscript individuals to his cause. 
Not direct mind-control per se, it is a subtle influence of the 
subconscious; individuals under the Mule’s influence behave otherwise 
normally—logic, memories, and personality intact.” 


Scott Adams more blandly calls Trump a “master persuader” 
(http://blog.dilbert.com). The effect is the same, especially if 
Trump wins. Which | fear. And, hate to say, expect. 


Scott was right, and that surely makes Trump a much more 
interesting case study than Clinton, whose only quotable line during 
the whole campaign was “basket of deplorables”. Her relatively 
blah public persona also made her a dressmaker’s dummy for every 
adjectival characterization Trump chose to clothe her in, just as he 
had done with the most threatening of the 17 Republican opponents 
he knocked off in the primaries (examples: “low-energy Jeb”, “little 
Marco”, “lyin' Ted”). Looking at the Heartbeat Al research, it seems 
the one that stuck in the final round was “Crooked Hillary”. 

The high degree to which Trump was disliked, even by people 
who voted for him, is nowhere in any of the prevailing narratives in 
election coverage, whether by (what's left of) mainstream media or 
by the millions yakking about it on Facebook and Twitter. Instead, 
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Google Trends Interest over time influencer 
Search term 


Figure 2. Google Trends Results for “Influencer” 


we're mostly hearing about how those deplorables won respect by 
defeating “the elites”. 

Speaking of which, nearly all major newspapers sensibly opposed 
Trump, and most endorsed Clinton. Even some papers that normally favor 
Republican candidates either endorsed Clinton or suggested voting for 
anyone but Trump. In its 126-year history, the Arizona Republic never 
endorsed a single Democrat, but inveighed in favor of Hillary Clinton for 
this election (http://www.azcentral.com/story/opinion/editorial/20 16/09/27/ 
hillary-clinton-endorsement/91 198668), calling Donald Trump a “two-bit 
billionaire” (http://www.azcentral.com/story/opinion/editorial/2016/01/29/ 
donald-trump-republican-imposteropinioneditorial/79417514) and much worse. 

Wikipedia's list of Republicans opposed to Trump 
(https://en.wikipedia.org/wiki/List_of_Republicans_who_opposed_ 
Donald_Trump_presidential_campaign,_2016) and influencers in favor 
of Clinton (https://en.wikipedia.org/wiki/List_of_Hillary_Clinton_ 
presidential_campaign_endorsements,_2016) were both very long 
and thick with highly notable people and institutions. None of them 
mattered. Trump won anyway. 

So much for the influence of influencers—or at least influencers of the 
usual kind. And it Is interesting that this is all happening In a time when 
the term “influencer” is in marketing vogue (https://Awww.google.com/ 
trends/explore?date=all&q=influencer). See Figure 2. 

You'll see almost exactly the same result for “influencer marketing” 
(https:/Awww.google.com/trends/explore?date=all&q=influencer%20marketing), 
meaning that the main interest in “influencers” is among marketers. 
Not surprisingly, if you look up “influencer marketing”, you'll 
get a load of results (nearly a half-million in my case, at that 
link), nearly all of it pitching shortcuts to earning influence. Says 
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But I think there’s more going on here. Before 
the internet came along, sources of authority 
were clear and understood. 


Wikipedia (https://en.wikipedia.org/wiki/Influencer_marketing): 


Influencer marketing (also influence marketing) is a form of 
marketing (https://en.wikipedia.org/wiki/Marketing) in which 

focus is placed on specific key individuals (or types of individual, 
https://en.wikipedia.org/wiki/Individual) rather than the target market 
as a whole. It identifies the individuals that have influence over 
potential buyers (https://en.wikipedia.org/wiki/Buyer), and orients 
marketing activities around these influencers. 


Influencer content may be framed as testimonial advertising where 
they play the role of a potential buyer themselves, or they may be 
third parties. These third parties exist either in the supply chain 
(retailers, manufacturers, etc.) or may be so-called value-added 
influencers (such as journalists, academics, industry analysts, 
professional advisers, and so on). 


As a journalist, I'm often targeted for manipulation by influence 
marketers. While | ignore or reject those appeals, | am also struck by 
the ironic sense that my own influence has been going down over 
recent years, roughly on a path that seems to mirror the upward 
curve in the graph shown in Figure 2. | am also sure the same Is true 
for most journalists, if not all of them. Celebrities too. Endorsement 
failings in this election are just one example of that decline at work. 

But | think there’s more going on here. Before the internet came 
along, sources of authority were clear and understood. We had 
scientists, researchers, think tanks and other confirming or denying 
sources for claims, findings and stories told by politicians, business 
folk and other suspects given to lying and exaggeration. Good 


111 | January 2017 | http://www.linuxjournal.com 


EOF 


journals and journalists could be trusted to find and expose essential 
facts, regardless of who got offended. Many fewer journalists are 
employed now, and many surviving journals have either gone out 

of business or morphed into the “content production” business. 
Most commercial publications (ours included) now show ads that 

are annoying, based on unwelcome tracking, or both, which reduces 
readership while eroding trust. Meanwhile, the old “mainstream 
media” are now just waves atop an ocean of “content” generated by 
anything and anybody. 

Many journalists took great heart in the promise of the net and 
the web, back in the early years of both—especially how it gave 
everybody their own platform for publication and participation. In 
We the Media (https://www.amazon.com/We-Media-Grassroots- 
Journalism-People/dp/0596102275), first published in 2004, Dan Gillmor 
(https://dangillmor.com) details the progress of “journalism as lecture to 
journalism as conversation or seminar”. For example, take a post | put 
up on October 23, 2003, titled “Listen and Learn” (http://doc.weblogs.com/ 
2003/10/22#listenAndLearn). There | visited some of my own learning 
experiences with journalism’s evolution in real time. Here’s an excerpt: 


| find myself thinking there are three approaches to journalism 
represented here. One is the “cool” approach of traditional journalism, 
including network broadcasting (in which NPR is no exception). One 

is the “hot” approach of talk radio, which has since expanded to TV 
sports networks and now Fox TV. The third is the engaged approach 

of weblogging. What we're doing here may be partisan in many cases, 
but it is also inconclusive. Blogging is about making and changing 
minds. It’s less about scoring points against perceived enemies—with 
certain exception (http://www.rageboy.com/blogger.html), of course— 
than about scaffolding new and better understandings of one subject 
or another. 


Jay Rosen (https://twitter.com/jayrosen_nyu), a journalism professor 
at NYU, turned a one-liner from that passage into the title for a 
follow-up post: “Blogging is About Making and Changing Minds” 
(http://archive.pressthink.org/2003/10/23/doc_inconclusive.html). He 
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concludes that post with this: 


The cool, neutral, professional style in journalism says: get both sides 
and decide for yourself. The hotter, more partisan press says: decide for 
yourself—which side?—then go get information. The weblog doesn’t 
want to be either of these, but it checks and it balances both. 


Those days are so gone. 

So let's return to the noise—by which | mean Facebook. 

At issue as | write this is “fake news” on Facebook, which seems to have 
influenced a lot of people in the election—or so says Craig Silverman in 
BuzzFeed (https:/Awww.buzzteed.com/craigsilverman/viral-fake-election-news- 
outperformed-real-news-on-facebook?utm_term=.mijmqmeLNG#.qylPQP34d1), 


Total Facebook Engagements for 
Top 20 Election Stories 


15 million 

12 million 
9 millon 8.7 million 
6 millon 7.3 million 


3 millon 
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ENGAGEMENT REFERS TO THE TOTAL NUMBER OF SHARES, REACTIONS, AND COMMENTS 
FOR A PIECE OF CONTENT ON FACEBOOK SOURCE: FACEBOOK DATA VIA BUZZSUMO 


Figure 3. Influence of “Fake News” on Facebook (from Craig Silverman) 
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king of what we might call the “newstream” media. Craig did a bunch 
of research, most dramatically represented by the graphic shown in 
Figure 3. 

| could go on, but I'd rather not, since by now I've spent more time 
on this piece than on anything I’ve written here in many years. (And 
I've been doing that since 1996.) 

Here’s the Linux connection: we need to hack news back ina 
logical direction, and away from the fact-free, misleading and 
emotion-stirring ways that news is made today. The mainstream 
media is beyond fixing. So is the newstream media, so long as it 
remains dependent on surveillance-based advertising, clickbait and 
fake news of its own. 

| don’t know how we do that, but we've hacked the world before: 
with free software, Linux and open source, just to name the Big Three. 

Time to do it again. = 
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Tame the 
Docker Life 
Cycle with 
SUSE 
HINUK By John S. Tonello 


It’s no accident or mere passing fad that 
containers are revolutionizing how IT 
shops of all sizes do their work. Whether 
you're looking to make better use of 
existing data-center resources or improve 
portability to the cloud, Docker and 

the new-found freedom it offers to use 
virtual environments for everything from 
development to enterprise applications 
holds a lot of promise. 

The challenge Is figuring out how best 
to move beyond a standard Docker install 
to an enterprise-worthy solution that’s 
secure, easy to manage and scalable. It’s 
also important to find ways to manage all 
your containers easily as well as the images 
you modify and plan to reuse. After all, 
containers are only part of any enterprise, 
which is now a healthy mix of bare-metal 
boxes, virtual machines, containers and 
on- and off-premises clouds. Tools that can 
help provide a common framework—and 
familiar interfaces—are critical. 

With SUSE Enterprise Linux Server 
12 and the tools it offers, you and your 
team can begin to solve real-world 
problems, tame the Docker life cycle, 
and create, run and maintain containers 
at nearly any scale. 


The Container Revolution 
Anyone managing hardware—from a 
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Tame the Docker Life 
Cycle with SUSE 


few blades to full data centers—knows 
that bare-metal server deployments are 
costly, time-consuming and not very 
efficient. Even if you could still afford it, 
the idea of running one or two services 
on a single physical server—maybe a 
database here, a website there—is just 
not practical. Even if you’re the best 
system administrator out there, you 
can really make only educated guesses 
about the maximum amount of CPU, 
memory and storage a particular service 
will need over time. Once you do the 
math and purchase the hardware, you 
know there surely will be hours, days 
and weeks when your physical server's 
capacity is idle and of no use to you. 
Virtual machines changed all that 
by enabling more efficient use of that 
same physical server's resources by 
sharing them across separate instances 
of Linux and Windows servers. With 
the advent of VMware and Hyper-V and 
open-source KVM and Xen, suddenly 
you could place multiple servers on 
a single physical box, quickly move 
them between clusters, more easily run 
backups and restores, clone them and 
manage them all from a single interface. 


To continue reading, download the complete eBook 


for FREE at http://geekguide.linuxjournal.com. 
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“@ macromaker.module x macromaker js d % Jquery.insertArCaret.js 
3 * 

4 * Usage: $(obj). insertAtCursor( text); 

5 * obj = a textarea or textfield 

6 * text = a string to insert 

7 */ 

8 $.fn.extend({ 

So insertAtCaret: function(myValue){ 

190 return $(this).each(function(){ 

11 // If target element is hidden, don't do it. 

12 if ($Cthis).isC' :hidden') || $c: ).parents(' :hidde 
13 return; 

14 } I 

15 if C(document.selection) { 
this. fo 
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